1. EDUCAUSE 2002 Service Quality Assessment in a Digital Library Environment Service Quality Assessment in a Digital Library Environment Atlanta, Georgia October 3, 2002  Duane Webster

2. Overview of Discussions 1.New Models for Understanding and Describing Library Success 2.ARL’s New Measures Initiative 3.LibQUAL+ Project Development 4.Experience with LibQUAL+ 5.Conclusions and Next Steps

3. The Association of Research Libraries www.arl.org A SSOCIATION OF R ESEARCH L IBRARIES Mission:Shaping and influencing forces affecting the future of research libraries in the process of scholarly communication. Members:123 major research libraries in North America. Ratios:4% of the higher education institutions providing 40% of the information resources. Users:3 million students and faculty served. Expenditures:$2.35 billion annually, $727 million for acquisitions of which 9% is invested in access to electronic resources.

4. ARL New Measures Initiative uCollaboration among member leaders with strong interest in this area uSpecific projects developed with different models for exploration uIntent to make resulting tools and methodologies available to full membership and wider community

5. LibQUAL+™ Description LibQUAL+ TM is a research and development project undertaken to define and measure library service quality across institutions and to create useful quality- assessment tools for local planning.

6. Project Resources LibQUAL+ TM is an ARL/Texas A&M University joint effort. The project is supported in part by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) and a 3-year grant from the National Science Foundation (NSF)

7. The Imperative for our Research In an age of accountability, there is a pressing need for an effective and practical process to evaluate and compare research libraries. In the aggregate, among the 122 Association of Research Libraries (ARL) alone, over $2.8 billion dollars were expended in 1999/2000 to satisfy the library and information needs of the research constituencies in North America (Kyrillidou & Young, 2001, p. 5).

8. The Problem of Assessment in Research Libraries uThe lack of metrics uARL Membership Criteria Index variables emphasize inputs, primarily expenditures uNo demonstrable relationship between expenditures and service quality uTo rise in the ARL Index it is only necessary to spend more

9. Assessment “The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two.” Sarah Pritchard, Library Trends, 1996

10. LibQUAL+™ Project Goals uEstablishment of a library service quality assessment program at ARL uDevelopment of web-based tools for assessing library service quality uDevelopment of mechanisms and protocols for evaluating libraries uIdentification of best practices in providing library service

11. LibQUAL+ TM Participants Year 1 Year 2 Year 3 Spring 2001Spring 2000Spring 2002 12 Participants 43 Participants 164 Participants For More Information about Participants: Visit the LibQUAL+ web site.

12. Relationships: perceptions, service quality and satisfaction ….only customers judge quality; all other judgments are essentially irrelevant” Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

13. Source: Parasuraman, ARL Symposium on Measuring Service Quality Washington, DC, October 2000

14. 70+ Interviews conducted 70+ Interviews conducted   York University  University of Arizona  University of Connecticut  University of Houston  University of Kansas  University of Minnesota  University of Pennsylvania  University of Washington  Smithsonian  Northwestern Medical

15. Atlas TI Dimensions of Analysis

16. “By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.” Graduate student Self-reliance

17. “…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.” Faculty member Self-reliance

18. Affect of Service “I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.” Faculty member

19. Comprehensive Collections “I think one of the things I love about academic life in the United States is that as a culture…, we tend to appreciate the extraordinary importance of libraries in the life of the mind.” Faculty member

20. Comprehensive Collections “I sense that if I were in an institution that didn’t have the rich collections as this library and the very effective staff members that this library has that I would imperceptibly slip in my discipline….” Faculty member

21. Ubiquity of Access “Over time my own library use has become increasingly electronic. So that the amount of time I actually spend in the library is getting smaller and the amount of time I spend at my desk on the web … is increasing.” Faculty member

22. “You put a search on a book and it’s just gone; it’s not reacquired. … There’s more of a problem of lost books, of books that are gone and nobody knows why and nobody’s doing anything about it.” Faculty member Reliability

23. Reliability “I put something on reserve. And it didn’t show up, and somebody complained. I went back and said ‘I’ve asked for this to be put on reserve’ and they had lost the form. So I had to do it again.” Faculty member

24. “I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.” Faculty member Library as Place Library as Place

25. “The poorer your situation, the more you need the public spaces to work in. When I was an undergraduate, I spent most of my time in the library, just using it as a study space.” Faculty member

26. Library as Place “One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.” Faculty member

27. Dimensions of Library Service Quality

28. Affect of Service uEmerged as the dominant factor early in our work uAbsorbed several of the original SERVQUAL questions measuring Responsiveness, Assurance and Empathy uIn the current analysis also includes Reliability uAll in all: the Human Dimension of Service Quality

29. Library as Place uTranscends the SERVQUAL dimension of Tangibles to include the idea of the library as the campus center of intellectual activity uAs long as physical facilities are adequate, library as place may not be an issue

30. Personal Control uHow users want to interact with the modern library uPersonal control of the information universe in general and web navigation in particular

31. Access to Information uUbiquity of access: information delivered in the format, location and time of choice uComprehensive collections

32. u42 — ARL Libraries u35 — Health Sciences Libraries u36 — State Colleges & Universities (excluding ARL) u34 — Private Colleges & Universities (excluding ARL) u15 — Community Colleges u 2 — Special & Public Libraries (Smithsonian & NYPL) LibQUAL+  2002 Iteration

33. The Challenge of Analysis uThere are few, if any useful conclusions to be drawn from aggregate data of all institutions, because their missions and subsequent user expectations for service are too diverse. uThere are commonalities in service delivery profiles that merit further investigation. uIn the long run, information that may be derived from demographic responses of individuals may yield the richest data.

34. Two Interpretation Frameworks uScore Norms uZone of Tolerance

35. Zone of Tolerance uThe area between minimally acceptable and desired service quality ratings uPerception ratings ideally fall within the Zone of Tolerance

36. Aggregate Dimension Summary (n=70,445) Note: LibQUAL+  Spring 2002 Aggregate Survey Results. (2002). vol. 1, p. 24

37. Note: LibQUAL+  Spring 2002 Aggregate Survey Results. (2002). vol. 2, p. 40 Faculty Item Summary

38. Score Norms uNorm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample. uLibQUAL+ TM norms have been created at both the individual and institutional level

39. ARL Top 40 6.84 (.46) ARL Other 6.74 (.27) Private Colleges 6.90 (.49) State Colleges & Universities 6.38 (.30) Community Colleges 7.26 (.55) Overall Mean Scores and Service Adequacy Gap Scores By Cohort Group 2002 LibQUAL+  Iteration (n=162) AAHSL 7.07 (.56)

40. LibQUAL+™ Fundamental Contributions to the Measurement of Effective Delivery of Library Services uShift the focus of assessment from mechanical expenditure-driven metrics to user-centered measures of quality uRe-ground gap theory for the library sector, especially academic libraries uGrounded questions yield data of sufficient granularity to be of value at the local level uDetermine the degree to which information derived from local data can be generalized, providing much needed “best practices” information uDemonstrate the efficacy of large-scale administration of user-centered assessment transparently across the web uMakes little demand of local resources and expertise

41. Recognize the limitations of listening to customers uCustomers have a limited frame of reference and tend to offer incremental, rather than bold, suggestions –A better slide rule –The microwave oven, Post-it Notes, Velcro uInnovation is the responsibility of staff Anthony W. Ulwick, Harvard Business Review, January 2002

42. Shift the focus to outcomes uPlan outcome-based customer interviews uCapture desired outcomes uOrganize the outcomes uRate the outcomes for importance and satisfaction –Opportunity algorithm: (Importance+(Importance-Satisfaction)=Opportunity) uUse the outcomes to jump-start innovation Anthony W. Ulwick, Harvard Business Review, January 2002

43. When desired outcomes become the focus of customer research, innovation becomes a manageable, predictable discipline. Anthony W. Ulwick, Harvard Business Review, January 2002

44. Summary uSurvey can handle large numbers uSurvey can be turned around quickly uLimited local expertise required uInterpretations should be across chosen cohorts uLots of opportunities for using demographics to discern user behaviors uQ-technique and other tests will provide opportunities to observe how institutions may cluster

45. LibQUAL+  Related Documents LibQUAL+  LibQUAL+  Web Site http://www.arl.org/libqual/ LibQUAL+  LibQUAL+  Bibliography http://www.coe.tamu.edu/~bthompson/servqbib Survey Participants Procedures Manual http://www.arl.org/libqual/procedure/lqmanual2.pdf

46. NDSL LibQUAL+ Activities 120-200 qualitative interviews to contribute to identifying dimensions of digital library service quality Test and refine dimensions of digital library service quality and self-sufficiency through development of total market survey Implement survey across variety of organizational and digital library implementations

47. NDSL LibQUAL+ Goals Define dimensions of digital library service quality from the users’ perspectives Develop tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts Identify digital library best practices that permit generalizations across operations and development platforms

48. NSF Grant Assess service quality in digital libraries 3 year period Adopt LibQUAL+ instrument for use in the Science, Math, Engineering and Technology Education Digital Library community (NSDL)

49. LibQUAL+ TM  Project History Sept. 1999 Jan2000 July 2000 Sept. 2000 Oct. 2000 Jan. 2001 June 2001 9/99 - ARL launches “New Measures Initiative” which includes the study of service effectiveness known as SERVQUAL spearheaded by Texas A&M University. 1/00 - Initial 12 institutions begin the SERVQUAL study led by the Texas A&M team. 7/00 - LibQUAL+ as a distinct library-based assessment tool is presented. 9/00 - ARL and Texas A&M awarded a FIPSE grant to fund further development of the LibQUAL+ project. 10/00 - The ARL symposium, “New Culture of Assessment in Academic Libraries Measuring Service Quality” attracts a group of 170 people. 1/01 - Representatives from 43 research and university libraries participating in the Spring 2001 Implementation meet in Washington, DC during ALA midwinter. 6/01 - National Science Foundation awards grant to ARL and Texas A&M to adapt LibQUAL+ for NSDL

50. LibQUAL+ TM Team ARL Duane Webster Martha Kyrillidou Kaylyn Hipps Julia Blixrud Jonathan Sousa Consuella Waller TAMU Fred Heath Colleen Cook Bruce Thompson Yvonna Lincoln Trey Thompson Julie Guidry

51. Overview of Discussions 1.New Models for Understanding and Describing Library Success 2.ARL’s New Measures Initiative 3.LibQUAL+ Project Development 4.Experience with LibQUAL+ 5.Conclusions and Next Steps

52. The Problem of Assessment in Research Libraries ARL Membership Criteria Index variables emphasize inputs, primarily expenditures To rise in the ARL Index it is only necessary to spend more No demonstrable relationship between expenditures and service quality The lack of metrics describing performance