1. Mixed Methods Not Mixed Messages
Improving Libguides with Student Usability Data
Nora Almeida, Instruction & Reference
Junior Tidal, Web Services
New York City College of Technology Library
City University of New York
Image: Rich Anderson, CC BY-SA 2.0

2. New York City Tech City University of New York (CUNY) Senior College
Undergraduate & Professional Degree programs
Commuter campus
Downtown Brooklyn “tech triangle”
Enrollment ~17,280
Student focused library
City Tech
Senior College at CUNY (largest urban public university in USA)
Undergraduate & Professional Degree programs
Commuter campus
Downtown Brooklyn “tech triangle”
Enrollment ~17,280
Ursula Schwerin Library
Student focused (culture / collection)
Robust instruction program outpacing staffing (credit courses & one shots)
Internally managed website w. Systems & web services librarians
Like many colleges, we need more investment in digital resources for students (commuter culture / hybrid & online courses) & to fill instruction gaps

3. Libguides Popular research guide platform Springshare
Consortial license in 2015
Migrated from MediaWiki to LibGuides v. 2

4. Defining the Problem Guides mapped over from MediaWiki quickly
Lack of consistency across guides
Lots of words & not a lot of interactivity or
embedded search opportunities
Duplication of content across guides & inconsistent use of links
City Tech librarians inexperienced with platform
Guides mapped over from MediaWiki w/out sufficient redressing of design & usage scenarios
Lack of consistency across guides in terms of nav, layout
Lots of words & not a lot of interactivity or
embedded search opportunities
Duplication of content across guides & inconsistent use of links
City Tech librarians inexperienced with platform
Image: Andy Bright, CC BY-SA 2.0

5. Study Objective Understand student design preferences Navigation
Layout
Terminology
Extent of content displayed
Learn about students’ impressions of multimedia, text-based, & interactive instructional content
Develop best practices documentation
Hardcode a LibGuides templates & create widgets
Image: Andy Bright, CC BY-SA 2.0

6. Methodology Paper prototyping Advanced scribbling Task Analysis
The multi-method approach provided additional insights and the combined interpretation explained the partly contradictory findings and avoided several misinterpretations of the paper prototyping data. Furthermore, the advanced scribbling and the handicraft task provided additional feedback on specific elements and the structure of the planned redesign.
—Linek & Tochtermann (2015)
Paper prototyping
Advanced scribbling
Task Analysis
Think Aloud
Semi Structured Interviews
Methodology adapted from 2015 study published by Stephanie B. Linek & Klaus Tochtermann in Forum: Qualitative Social Research
In their study: point to mixed methods as a strategy to “lower barriers” to critique--important given that research indicates that UX study participants feel pressure to impress the experimenter
Smaller participant pool needed with mixed methods
Mixed bag:
Paper prototyping
Advanced scribbling
Task Analysis
Think Aloud
Semi Structured Interviews

7. Process Methodology & Design IRB Approval Recruitment
Testing & Analysis
Best Practices Document
Libguide Templates
Junior (web services) / Win Sea (instructional design intern) / Nora (instruction)
Study Design
Survey of lit
Refinement of methods
Prototype creation (iterative)
Interview questions
Handouts
IRB
Recruitment (10 students):
Testing & Analysis (Fall 2016)
3 administer test: 1 proctor / 1 note taker / 1 human computer
Best Practices Documentation (March 2017)
Templates (Summer 2017)

8. Prototyping Paper prototypes
Created prototypes using adobe illustrator
Looked at existing guides and identified “typical” guide (layout & nav) to use as our control
Created a variant w/ different nav / more interactivity (pictured)

9. Scribbling Instructions for Advanced Scribbling
Take a close look at these two paper prototypes and use the pens to evaluate the content, labels, and design elements on this research guide for English.
Use the green pen to mark things that are important
Use the yellow pen to mark things that are confusing
Use the red ped to mark things that are unnecessary
The advanced scribbling component allows users to identify what they like or dislike with colored highlighters. Participants write or scribble on the paper layouts. Green = important elements, yellow = confusing elements, and red = unnecessary elements. Also a pen to make notations.
Student conduct scribbling on BOTH prototypes before interview takes place in order to allow for critical reflection of content and avoid skewing student responses

10. Sample Interview Questions
Which research guide do you prefer?
What don’t you like about the guide you didn’t select?
Which features of the research guide made you select this guide?
How much content do you think should be included on a research guide?
What labels or features did you find confusing?
Do the menu labels on this guide make sense to you?
Semi-Structured
Questions about:
Design
Extent of content
Labels
But also:
Contextual questions about experience w/ research etc.
Learning style / media preferences

11. Research Task
You’re writing a paper on post-colonial Caribbean literature. Your professor assigned you to use scholarly articles and books from the library to back up your thesis.
Where would you go on this guide to find scholarly articles?
Where would you go on this guide to search for books in the City Tech library’s collection?
Completed “task” on prototype they preferred
Encouraged students to “think aloud” throughout
Took notes: number of clicks / Success Failure

12. Participant Profiles Students:
Some exposure to academic research (most characterized themselves as beginners or intermediate researchers)
Majority are not active library users / at least not academic libraries (see graphic)
4 out of 10 have attended a library instruction session either here or at a college they attended before
None of the students had ever used a research guide
The majority of students expressed a preference for multimedia (audio / video) instructional content

13. Prototype A 50% students preferred prototype A
50% students preferred prototype B
*The Linek / Tochtermann study that we based our methodology on also found “no clear preference for a design version” illustrating that paper prototyping A/B testing alone may not yield fruitful results at least without a large sample size.
By the numbers...
Important Elements
A Tab labels (80%) - Menu for guide navigation
A Ask A Librarian (70%) - Contacting a librarian via /chat/phone
A Welcome Message (50%) - Context for guide (majority of students who favored this element prefered text based learning modalities)
Confusing & Unnecessary Elements
A Class exercise - 50% Confusing / 30% Unnecessary
A Welcome image - 10% Confusing / 20% Unnecessary (10% important but misread placeholder)

14. Design Elements FINDINGS--advanced scribbling
Paper Prototype A (Control) Summary

15. Prototype B Important Elements
B Nav (70%) - Nav itself important but scribbling & qualitative feedback reveals 50% found some labels confusing / 20% found some elements unnecessary
B Search box (60%) - search field for discovery layer
B Book carousel (70%) - Dynamic book carousel
B Video (40%) - Instructional Video
B Ask A Librarian (90%)
B Your Librarian (60%) - Biographical/contact information for guide author (subject liaison)
Confusing / Unnecessary Elements
B Video 30% unnecessary (reflecting learning modality preferences)
B Your Librarian 30% unnecessary (reflecting redundancy)
B SPECIFIC Tab Labels 70% unnecessary / confusing: Begin Search (viewed as duplication of search box) / Browse Web Resources
Students didn’t comment on or provide feedback on verb vs. noun in labels “Find Artices” vs. “Articles”

16. Design
Elements
Paper Prototype B (Redesign) Summary

17. Qualitative Feedback "Begin Search" should be "Home" Prototype A:
There are two "Search"
"should not be 2 search options"
Ask a librarian + contact [your librarian] are redundant
Combine "ask a librarian" and "your librarian" boxes
Prototype A:
Home page “lacks helpful content”
“Has more options of where to start looking”
Prototype B:
“I like that I can search”
[Referring to Navigation labels]: “Vague, Missing information”
Qualitative Feedback (Sonsteby & DeJonghe study parallels)
Reasons for selecting each guide:
A: Layout or Organization / Navigation & Labels / Welcome message & Ask a Librarian
B: Search / Variety of content types / Video
Reasons for not selecting each guide:
A: Boring / Not interactive / Purpose unclear
B: Video / Unclear or vague labels
Other Findings:
Language matters
Students want a clear “purpose”
Students don’t like duplication (Ask a Librarian / Your librarian & Search box / Begin Search)
Capacity to actually FIND information (as opposed to learning how to find info) via guide is key
Extent of content (how many boxes / how many pages)
Feedback varied:
Shallow depth ie. less clicks
Minimal or simple
Divide content by “type” or format

18. Task Analysis
Task 1: Where would you go on this guide to find scholarly articles?
Task 2: Where would you go on this guide to search for books in the City Tech library’s collection?
High Success Rate for both Prototypes
Since Prototype A (control) does not have a search box widget, some navigating outside of the guide is required which accounts for most of the failures

19. Navigation
Low click rate for both prototypes. Slightly lower average on prototype B reflects capacity to search from the homepage.
Lower click rate for task 2 may reflect learnability / student comfortability with the guide

20. Disparities
Advance scribbling data about primary navigation is misleading since qualitative feedback illustrates the problem is the labels not the design
Students had difficulty reading design cues (eg. confused about image placeholder & dynamic book carousel)
Several features were liked by some students but marked as confusing or unnecessary by others including:
Class exercise (Prototype A)
Introductory Text (Prototype B)
Welcome Text (Prototype A)
Video (Prototype B)
Advance scribbling data on navigation is misleading since qualitative feedback illustrates the problem is the labels not the nav design
Students had difficulty reading design cues (weakness of paper prototypes reflects limitations outlined in Sefelin, Tscheligi, & Gille)
Confused about image placeholder
Some students thought the book carousel would change to reflect search terms
Assumed certain elements were linked during tasks
Several features were liked by some students but marked as confusing or unnecessary by others
Reflecting personal design preference
Learning modality preferences

21. Conclusions
Positive correlation between modality preferences & design preferences
No correlation between research experience & design preferences
No “best” layout emerged
Focus on design elements ranked favorably across prototypes to inform hybrid design
Positive correlation between modality preferences & design preferences
Student who selected Prototype B: More likely to mark interactive / multimedia elements as important
Students who selected Prototype A: More likely to mark text based contextual elements as important
No correlation between research experience & design preferences (but we had a small sample / narrow range of research experience)
No “best” layout emerged
Focus on design elements ranked favorably across prototypes to inform hybrid design

22. Implications for Design
Best Practices → Template
Navigation / labels
Number of pages (5 default)
Search box w/ tips
Welcome text
Combined Ask a Librarian / Your Librarian box
Databases w/ Icons
Widgets
Video
Book Carousel
Takeaways based on data and qualitative feedback:
Maintain control prototype tab labels but include more multimedia / interactive content
Accommodate different learning modality preferences
Ensure students can actually complete basic research tasks from within guide
Regardless of modality preference / prototype selection Search Box important
Avoid redundancy
Make guide purpose explicit (Gimenez, Grimm, and Parker)
Other items like the book carousel / video may not be appropriate for every guide
and feedback from students was too ambiguous to justify inclusion in a template (instead optional assets / widgets)

23. Areas for Further Research
Learning modality preferences and design implications
Correlation between research experience and design preferences
Use contexts for digital instructional resources like LibGuides
Impact on student research outcomes / achievement
More data needed re: learning modality preferences and design (perhaps conduct a computer based or hi-fi UX test once template is implemented / include more questions / tasks about learning preferences)
Diversify study sample to include more advanced level researchers to see if broader sample yields different data
Consider use contexts for digital instructional resources like LibGuides. We had questions about:
Emphasis on discovery, information literacy (acculturation), or resource curation?
Flipped classrooms
Independent research assignments
As supplement / replacement for instruction
Use in one-shot vs. credit bearing research courses
Impact on student research outcomes / achievement
Image: Gerd Leonhard, CC BY-SA 2.0

24. Bibliography
DeSimio, T., & Chrisagis, X. (2014). Rethinking our LibGuides to Engage our Students: Easy DIY LibGuides Usability Testing and Redesign that Works. Retrieved from
Dumuhosky, Laura; Rath, Logan T.; and Wierzbowski, Kenneth R., "LibGuides Guided: How Research and Collaboration Leads to Success" (2015). Library Presentations. Paper 2. Retrieved from
Linek, S., & Tochtermann, K. (2015). Paper Prototyping: The Surplus Merit of a Multi-Method Approach. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 16(3).
Martin, B., & Hanington, B. M. (2012). 01 A/B Testing. In Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions (pp. 8-9). Beverly, MA: Rockport.
Nielsen, J. (2005, August 15). Putting A/B Testing in Its Place. Retrieved from
Patricia Gimenez, Stephanie Grimm, and Katy Parker, "Testing and Templates: Building Effective Research Guides" (September 25, 2015). Georgia International Conference on Information Literacy. Paper 37. Retrieved from
Sefelin, R., Tscheligi, M., & Giller, V. (2003, April). Paper prototyping-what is it good for?: a comparison of paper-and computer-based low-fidelity prototyping. In CHI'03 extended abstracts on Human factors in computing systems (pp ). ACM. Retrieved from
Sonsteby, A., & DeJonghe, J. (2013). Usability testing, user-centered design, and LibGuides subject guides: A case study. Journal of Web Librarianship, 7(1), doi: /
Young, S. W. (2014). Improving Library User Experience with A/B Testing: Principles and Process. Weave: Journal of Library User Experience, 1(1).

25. Questions? Nora Almeida nalmeida@citytech.cuny.edu @nora_almeida
Junior Tidal
@juniortidal