1. Studying Distance Students: Methods, Findings, Actions Fifteenth Distance Learning Services Conference Memphis, Tennessee April 20, 2012 Beth Avery and Diane Wahl University of North Texas

2. Introduction Assessment at UNT LibQUAL+® results Focus groups Ethnographic study Triangulation How we responded What we learned

3. Assessment of User Needs at UNT LibQUAL+® survey starting in 2005 Identifies Need input on how to improve/change

5. Total Spring 2012 Headcount

6. On Campus Students I don't use the physical library on campus, but the online resources are excellent. The only problem I have ever had is with journal articles only being available at the actual library. Commuting from Plano makes it difficult to access those articles as I am only on campus 1-2 nights per week and only get there from work just in time for class. Thank you for the valuable online resources.! [sic] I am very busy and generally do not have time to physically visit the on-campus library.

7. Cost of attendance Value for the price Merit scholarships available Extracurricular opportunities Availability of recreational facilities Cost of attendance Value for the price Merit scholarships available Extracurricular opportunities Availability of majors Quality of majors Quality of on- campus housing Cost of attendance Value for the price Availability of majors Quality of majors Cost of attendance Academic reputation Personal attention Quality of academic facilities Quality of on- campus housing Availability of majors Quality of majors Academic reputation Personal attention Quality of academic facilities Academic reputation Personal attention Quality of academic facilities Quality of on- campus housing Merit scholarships available Academic reputation Personal attention Quality of academic facilities Quality of on- campus housing Merit scholarships available Availability of majors Quality of majors Value for the price Enrollment Management, Spring 2012 Very Important & UNT Rated Higher Very Important & UNT Rated Lower Admitted Students’ Ratings of UNT on Characteristics that they consider “Very Important” 2011 2010 2009 2008

8. : What It Measures minimum desired perceived superiority gap adequacy gap Thermometer Chart

9. Recruiting Convenience sample Eliminating bias – Selection bias – Nonresponse bias

10. Distance Learners

11. Data Analysis PRIORITIES Item #Item Description2009 Rank IC-1Making electronic resources accessible from my home or office1(8.55) IC-4The electronic information resources I need2(8.52) IC-2A library Web site enabling me to locate information on my own3(8.49) IC-8Print and/or electronic journal collections I require for my work4(8.49) IC-6Easy-to-use access tools that allow me to find things on my own5(8.4) LARGEST SUPERIORITY GAPS Item #Item Description2009 Rank IC-2A library Web site enabling me to locate information on my own1(-1.43) IC-6Easy-to-use access tools that allow me to find things on my own2(-1.3) LQ-1Availability of online help when using my library’s electronic resources3(-1.27) IC-8Print and/or electronic journal collections I require for my work4(-1.17) IC-4The electronic information resources I need5(-1.15) LARGEST ADEQUACY GAPS Item #Item Description2009 Rank IC-2A library Web site enabling me to locate information on my own1(0) IC-8Print and/or electronic journal collections I require for my work2(0) IC-6Easy-to-use access tools that allow me to find things on my own3(0.05) LQ-1Availability of online help when using my library’s electronic resources4(0.19) IC-4The electronic information resources I need5(0.23)

12. Comment Analysis I am an online student and while the online library is probably very good (and probably deserves higher marks than I gave it above), I do not know how to use it and don't think that I've ever really been made very aware of it (until now). I have not been introduced to the library or library services. I have found that the internet and my company library give me what I need. By the way, I have yet to see a link to the libray [sic] on any of the pages I visit. It is not on web CT [sic] that I can see. In short, Where [sic] is it? If a person who is predominently [sic] an on-line [sic] student has to look for a library services link and the link to is not on the pages visited regularly then the library does not exist. It's the old saw, Location, Location, Location.

13. 2009/2011 Comparison LARGEST SUPERIORITY GAPS Item # Item Description 2011 Rank 2009 Rank LQ-4 Online course support (readings, links, references)1(-1.01) IC-6 Easy-to-use access tools that allow me to find things on my own2(-1)2(-1.3) LQ-3 Making me aware of library services3(-1) IC-2 A library Web site enabling me to locate information on my own4(-0.99)1(-1.43) IC-4 The electronic information resources I need5(-0.93)5(-1.15) LQ-1 Availability of online help when using my library’s electronic resources 3(-1.27) IC-8 Print and/or electronic journal collections I require for my work 4(-1.17) LARGEST ADEQUACY GAPS Item # Item Description 2011 Rank 2009 Rank LQ-4 Online course support (readings, links, references)1(-0.06) IC-1 Making electronic resources accessible from my home or office2(0.14) LQ-3 Making me aware of library services3(0.19) IC-5 Modern equipment that lets me easily access needed information4(0.24) IC-6 Easy-to-use access tools that allow me to find things on my own5(0.26)3(0.05) IC-2 A library Web site enabling me to locate information on my own 1(0) IC-8 Print and/or electronic journal collections I require for my work 2(0) LQ-1 Availability of online help when using my library’s electronic resources 4(0.19) IC-4 The electronic information resources I need 5(0.23)

14. Focus Groups: IRB Approval Required to use results for Produces a plan for how the research will be conducted PresentationsPublications

15. Focus Groups: Recruiting Blackboard Announcements Incentives Distance Location Cohort

16. Focus Groups: Recruiting Online Student Focus Group Invitation HOW TO VOLUNTEER: Send an email before June 12 to diane.wahl@unt.edu. SPONSORED BY: UNT Libraries WHAT & WHO WHEN THANKS!

17. Focus Groups: Demographics Basic demographics – 5 men, 11 women – Most over 30 – 1 African American; 1 Asian American; 14 Caucasian – 1 foreign student College – College of Business – College of Education – College of Information – College of Public Affairs and Community Service

18. Focus Groups: Technology

19. Focus Groups: Analysis Microsoft Office Access Microsoft Office Excel

20. Focus Groups: Follow Up Committees Outreach Services

21. Focus Groups: Quotes COB Male: How would I like to learn--right as a distance learner I wish there were more mention of the UNT Library on the main UNT page, or my UNT page, or in ….. Blackboard so that way I don’t have to go to any course to look at that library. It should just pop out right there. MLS Female: The only thing that I find out about these things that ya’ll have on campus like training for graduate students with RefWorks and graduate research. It seems like they are always on campus, and maybe ya’ll do have it where distance students can access that or get transcripts somehow. But, I’ve never seen it, but I would like to do something like that. But I just live too far away from Denton to try to actually go to one. Applied Technology Male: Well I will say this about the UNT Library. When I was going there in 1994 and 1995, um I actually had somebody lead me around and show me pretty much everything I needed to know at the physical library. But that was probably my most memorable experience was having somebody actually there with me. So that’s why I’m saying it would be great to have somebody online with me showing me.

22. Focus Groups: Recommendations Summer 2009 Online Student Focus Group Recommendations Increase the Libraries’ presence and visibility in Blackboard Make the UNT Libraries’ website easier to search Obtain, publicize, and provide instruction in the use of a discovery tool Provide library instruction that doesn’t require being at the physical library Communicate and collaborate more effectively with faculty members that teach online classes

23. Ethnographic Study: Anthropology Graduate Students Fall 2010 Graduate level Anthropology Dept. qualitative and ethnographic analysis classes Campus and online Used a variety of methods: – Focus groups – Interviews – Observations

24. Ethnographic Study: Recruiting Was difficult – What didn’t work – What was missing – What worked

25. Ethnographic Study: Demographics Represent 6 colleges and schools, including 3 with online degree programs Represent 6 colleges and schools, duplicating only 3 from the online student study

26. Ethnographic Study: Observations On campus students: In person at various physical locations on campus Distance students: Wimba and shared the desktop

27. Ethnographic Study: Interviews Biggest problem: Memory blips. Recommendations for the future – Conduct the interview soon! – More observations

28. Ethnographic Study: Recommendations Fall 2010 Anthropology Class Recommendations Improve user experience with the UNT Libraries website; (includes problems with access tools) Improve library instruction Improve communication

29. Triangulation Benefit: Addresses the issue of biases in social science research Types: – Data – Researcher – Theory – Method

30. Comparing Results of Different Methods for Online Students RankSpring 2009 LibQUAL+ Survey Items with Largest Superiority Gaps – Online Students 1A library Web site enabling me to locate information on my own 2Easy-to-use access tools that allow me to find things on my own 3Availability of online help when using my library’s electronic resources 4Print and/or Electronic journal collections I require for my work 5The electronic information resources I need Summer 2009 Online Student Focus Group Recommendations Increase the Libraries’ presence and visibility in Blackboard Make the UNT Libraries’ website easier to search Obtain, publicize, and provide instruction in the use of a discovery tool Provide library instruction that doesn’t require being at the physical library Communicate and collaborate more effectively with faculty members that teach online classes Fall 2010 Anthropology Class Recommendations Improve user experience with the UNT Libraries website; (includes problems with access tools) Improve library instruction Improve communication

31. Comparing Results of Different Methods for Online Students RankSpring 2009 LibQUAL+ Survey Items with Largest Superiority Gaps – Online Students 1A library Web site enabling me to locate information on my own 2Easy-to-use access tools that allow me to find things on my own 3Availability of online help when using my library’s electronic resources 4Print and/or Electronic journal collections I require for my work 5The electronic information resources I need Summer 2009 Online Student Focus Group Recommendations Increase the Libraries’ presence and visibility in Blackboard Make the UNT Libraries’ website easier to search Obtain, publicize, and provide instruction in the use of a discovery tool Provide library instruction that doesn’t require being at the physical library Communicate and collaborate more effectively with faculty members that teach online classes Fall 2010 Anthropology Class Recommendations Improve user experience with the UNT Libraries website (includes problems with access tools) Improve library instruction Improve communication RankSpring 2011 LibQUAL+ Survey Items with Largest Superiority Gaps – Online Students 1Online course support (readings, links, references) 2Easy-to-use access tools that allow me to find things on my own 3Making me aware of library services 4A library Web site enabling me to locate information on my own 5The electronic information resources I need

32. Actions Creation of a library presence in Blackboard Making subject librarians accessible to online students Collaboration with faculty to develop “library gateways to the content needed for the classes”

33. Actions (cont.) Federated search engine implemented Library web-site redesign with user input Library liaison/faculty relationships New position: Director of Communications and Marketing.

34. Lessons Learned Benefits to students Value of LibQUAL+® Use of standard surveys Antidote to survey burn out and timing

35. Lessons Learned (cont.) Importance of listening Access to a variety of viewpoints in small samples using non-probabilistic sampling College students and incentives Importance of the right technology

36. Lessons Learned (cont.) Benefits of unbiased perspective of outside researchers Importance of including groups likely to be required to make changes in planning change Triangulation of data achievable at low cost

37. Questions?