Global Developments in Assessment: LibQUAL+ ® and Beyond… 6th International JISC/CNI Conference York, U.K. July 6-7, 2006 Duane Webster Executive Director Association of Research Libraries Selena Lock Research and Development Cranfield University ARL

Duane Webster, Executive Director What are some of the current developments with library assessments efforts? ARL StatsQUAL ™ E-Metrics LibQUAL+ ® DigiQUAL ™ MINES for Libraries ™ Where are the most critical needs and opportunities? What are the lessons learned? Library Assessment in an Electronic Era

Duane Webster, Executive Director Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication. Members: 123 major research libraries in North America. Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources. Users: Three million students and faculty served. Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 31 percent is invested in access to electronic resources.

Duane Webster, Executive Director Thinking Strategically About Library Futures What is the central work of the library and how can we do more, differently, and at less cost? What important set of services does the library provide that others can’t? What new roles are needed? What advantages does the research library possess? What will be the most needed by our community of users in the next decade? How is user behavior changing? What should our libraries aspire to be ten years from now? What are the implications of technology driven change? What are the essential factors responsible for the success of the library?

Duane Webster, Executive Director Defining Success in a Digital Environment Crafting new measures of success. Moving from measuring inputs to outputs. Understanding impact of library roles and services. Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals. Reallocating and managing capabilities to focus on new definitions of success.

Duane Webster, Executive Director ARL Statistics and Measurement

Duane Webster, Executive Director. StatsQUAL ™ Duane Webster, Executive Director

Duane Webster, Executive Director StatsQUAL ™

Duane Webster, Executive Director Updating the Traditional ARL Statistics E-Metrics = ARL Supplementary Statistics –On going efforts to update and refine core data. –Exploring feasibility of collecting e-metrics. ARL Task Force on New Ways of Measuring Collections : –Growing concern with utility of membership index. –Study ARL statistics to determine relevance. –Develop Profile of Emerging Research Libraries.

Duane Webster, Executive Director E-Metrics: Recommended Statistics  Patron Accessible Electronic Resources (R1-3)  Use of Networked Resources and Services (U1-5)  Expenditures for Networked Resources and Related Infrastructure (C1-3)  Library Digitization Activities (D1-3)

Duane Webster, Executive Director E-Metrics: Recommended Statistics  Patron Accessible Electronic Resources (R1-3) R1 – Number of electronic full-text journals R2 – Number of electronic reference sources R3 – Number of electronic books

Duane Webster, Executive Director E-Metrics: Recommended Statistics  Use of Networked Resources and Services (U1-5) U1 – Number of electronic reference transactions U2 – Number of logins (sessions) to electronic databases U3 – Number of queries (searches) in electronic databases U4 – Items requested in electronic databases U5 – Virtual visits to library’s website and catalog

Duane Webster, Executive Director  Expenditures for Networked Resources and Related Infrastructure (C1-3) C1  Cost of electronic full-text journals C2  Cost of electronic reference sources C3  Cost of electronic books C4  Library expenditures for bibliographic utilities, networks and consortia C5  External expenditures for bibliographic utilities, networks and consortia E-Metrics: Recommended Statistics

Duane Webster, Executive Director Library Digitization Activities  Library Digitization Activities (D1-3) D1 – Size of library digital collection D2 – Use of library digital collection D3 – Cost of digital collection construction and management (Collecting these data requires staff familiar with the digital environment.)

Duane Webster, Executive Director A LibQUAL+ ® Update The LibQUAL+ ® premise, dimensions, and methodology LibQUAL+ ® results LibQUAL+ in action

LibQUAL+™ Presented by: Selena Lock Cranfield University 6th International JISC/CNI Conference York, U.K. July 6-7, 2006 A LibQUAL+ ® Update

A LibQUAL+ ® Update §The LibQUAL+ ® premise, dimensions, and methodology §LibQUAL+ ® results §LibQUAL+ ® in action

LibQUAL+™ The LibQUAL+ ® premise, dimensions, and methodology

The need for LibQUAL+ ® §Underlying need to demonstrate our worth §The reallocation of resources from traditional services and functions §Rapid shifts in information-seeking behavior Need to keep abreast of customer demands §Increasing user demands 37% of UK 16 – 18 year olds expect better libraries in return for their top-up fees

PERCEPTIONS SERVICE “….only customers judge quality; all other judgments are essentially irrelevant” Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press. The LibQUAL+ ® Premise

Multiple Methods of Listening to Customers §Transactional surveys* §Mystery shopping §New, declining, and lost-customer surveys §Focus group interviews §Customer advisory panels §Service reviews §Customer complaint, comment, and inquiry capture §Total market surveys* §Employee field reporting §Employee surveys §Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.

LibQUAL+ ® Development §An ARL/Texas A&M University joint developmental effort based on SERVQUAL §LibQUAL+ ® initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) §Initial project established a expert team, re- grounded SERVQUAL concepts, and designed survey methodology §Survey conducted at over 700 libraries resulting in a data base of over a million user responses

13 Libraries English LibQUAL+® Version 4000 Respondents QUAL QUAN QUAL QUAN QUAL PURPOSE DATA ANALYSIS PRODUCT/RESULT Describe library environment; build theory of library service quality from user perspective Test LibQUAL+® instrument Refine theory of service quality Refine LibQUAL+® instrument Test LibQUAL+ ® instrument Refine theory Unstructured interviews at 8 ARL institutions Web-delivered survey Unstructured interviews at Health Sciences and the Smithsonian libraries to survey administrators Web-delivered survey Focus groups Content analysis: (cards & Atlas TI) Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Vignette Re-tooling Iterative Emergent Libraries English, Dutch, Swedish, German LibQUAL+® Versions 160,000 anticipated respondents LibQUAL+® Project Case studies 1 Valid LibQUAL+® protocol Scalable process Enhanced understanding of user-centered views of service quality in the library environment 2 Cultural perspective 3 Refined survey delivery process and theory of service quality 4 Refined LibQUAL+® instrument 5 Local contextual understanding of LibQUAL+® survey responses 6

76 Interviews Conducted §York University §University of Arizona §Arizona State §University of Connecticut §University of Houston §University of Kansas §University of Minnesota §University of Pennsylvania §University of Washington §Smithsonian §Northwestern Medical

LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred

Dimensions of Library Service Quality Information Control Library Service Quality Model 3 Self-Reliance Equipment Timeliness Ease of Navigation Convenience Scope of Content Affect of Service Library as Place Reliability Assurance Responsiveness Empathy Refuge Symbol Utilitarian Space

LibQUAL+ ® Participants

Rapid Growth in Other Areas §Languages American English British English French Dutch Swedish §In development Chinese Greek Spanish German §Consortia Each may create 5 local questions to add to their survey §Types of Institutions Academic Health Sciences Academic Law Academic Military College or University Community College European Business Hospital Public State §Countries U.S., U.K., Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia

Participating Libraries World LibQUAL+ ® Survey 2005

LibQUAL+ ® Languages Over 700 institutions 1,000,000 respondents

Survey Instrument – “22 items…

…and a Box” §Why the Box is so Important: About 40% of participants provide open- ended comments, and these are linked to demographics and quantitative data Users elaborate the details of their concerns Users feel the need to be constructive in their criticisms, and offer specific suggestions for action

LibQUAL+™ LibQUAL+ ® results

Understanding LibQUAL+ ® Results §For the 22 items LibQUAL+ asks users’ to rate their: Minimum service level Desired service level Perceived service performance §This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings §Perception ratings ideally fall within the Zone of Tolerance

Key to Bar Charts Key to Radar Charts

LibQUAL+ ® 2005 Summary Colleges or Universities (American English)

LibQUAL+ ® 2005 Summary Colleges or Universities (British English)

Longitudinal Analysis for Colleges or Universities (American English)

Longitudinal Analysis for Colleges or Universities (British English)

Comparisons by user group College or University (British English)

General findings §Highly desired Making electronic resources accessible from my home or office Print and/or electronic journals I require for my work A haven for study, learning or research §Lowest Library staff who instil confidence in users Giving users individual attention Space for group learning and group study

LibQUAL+™ LibQUAL+ ® in Action

UK University Case Study – Using LibQUAL+ ® Results §Strategic Service Developments Data to support service development Ability to identify where not meeting expectations Measure if change has met need §Budget Discussions Data to support bid for increased funding Data to support case for change in emphasis (towards e-provision) §Marketing Position Status of the library within the University Importance of national & international benchmarking

LibQUAL+ ® Outcomes Funding received for: §New Web Services Administrator §Increased opening hours Now providing 222,578 seat hours per week §Library refurbishment programme reinstated at costs in excess of £8 million

Library Refurbishment From:To:

Why use LibQUAL+ ® ? Feedback from LibQUAL+ ® Users “Why did you choose to use LibQUAL+ ® ?” §LibQUAL+ ® was recommended to us as offering a well designed, thoroughly Library-focused set of survey tools §Opportunity to benchmark §Cost-effectiveness §Automated processing & fast delivery of results §Respectability and comparability (with others and historically)

The benefits of LibQUAL+ ® LibQUAL+ ® has enabled us to find out what a broad range of our users thought of the services we offer; what level of service- delivery quality we had achieved in their eyes, and to get a clear picture of what they actually wanted the Library to deliver (as opposed to what we thought they wanted). UK HE Institution, 2006

In Closing LibQUAL+ ® §Focuses on success from the users’ point of view (outcomes) §Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense §Requires limited local survey expertise and resources §Analysis available at local, national and inter-institutional levels §Offers opportunities for highlighting and improving your status within the institution §Can help in securing funding for the Library

LibQUAL+ ® Resources §LibQUAL+ ® Website: §Publications: §Events and Training: §LibQUAL+ ® Bibliography: §LibQUAL+ ® Procedures Manual:

Acknowledgements §Fred Heath, Bruce Thompson, Colleen Cook, Martha Kyrillidou and the rest of the LibQUAL+ ® team §Stephen Town, Cranfield University §Helen Durndell and Jacqui Dowd, Glasgow University §All SCONUL LibQUAL+ ® Participants and the SCONUL Working Group on Performance Improvement

LibQUAL + ® Contact Information §Martha Kyrillidou Director, ARL Statistics and Measurement Program §Selena Lock Research and Development, Cranfield University

Developing the DigiQUAL ™ Protocol for Digital Library Evaluation DigiQUAL™ NSF Funding Building on the LibQUAL+ ® experience Secures feedback on user’s perceptions of library’s web site Five questions on services, functionality, and content Goal is to determine utility, reliability, and trustworthiness

Duane Webster, Executive Director Background: ServQUAL  LibQUAL+ ®  DigiQUAL ™ LibQUAL+ ® Dimensions of Service Quality: Affect of Service Information Control Library as Place Developing DigiQUAL ™ Survey Items DigiQUAL™ 12 themes of service quality: Accessibility Navigability Interoperability Collection building Resource Use Evaluating collections DL as community for users DL as community for developers DL as community for reviewers Copyright Role of Federations DL Sustainability

Duane Webster, Executive Director Pilot Testing Survey Items and Implementation

Duane Webster, Executive Director Building a Survey Review and select items Issue: aligning items to individual DL needs & users – vocabulary and content

Duane Webster, Executive Director Customize Survey Issue: Flexibility vs. Standardization Building a Survey, cont.

Duane Webster, Executive Director Implementing Survey – Notification Methods Links on site Newsletters Next to resources Issues: no pop-ups, no individual s

Duane Webster, Executive Director Implementing Survey - Incentives Issues: must be easily transferable, requires address - clear IRB

Duane Webster, Executive Director (mis)Interpreting results from individual digital libraries in the context of other sites Sites reluctant to share data and results Analysis and Reporting - Issues

Duane Webster, Executive Director Outstanding Issues and Challenges Unique DLs: niche market, critical mass, both? Balance: – custom vs. generic content  results – flexible vs. standard implementation  scaling Mixed methods – Preserving user privacy – Collecting truly useful data Moving target: digital libraries as… it depends.

Duane Webster, Executive Director Assessing the Value of Networked Electronic Services: Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™ The MINES survey

Duane Webster, Executive Director A research methodology consisting of a web-based survey form and a sampling plan. Measures who is using electronic resources, where users are located at the time of use, and their purpose of use. Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E- Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+ ®, DigiQUAL ™ ). What is MINES?

Duane Webster, Executive Director How extensively do sponsored researchers use the new digital information environment? Are researchers more likely to use networked electronic resources from inside or outside the library? Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on- campus, but not in the library; or off-campus)? What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys? Are particular network configurations more conducive to studies of digital libraries patron use? Questions Addressed

Duane Webster, Executive Director A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library. Random moment/web-based surveys are employed at each site. Participation is usually mandatory, negating non- respondent bias, and is based on actual use in real-time. Libraries with database-to-web gateways or proxy re- writers offer the most comprehensive networking solution for surveying all networked services users during survey periods. MINES Methodological Considerations

Duane Webster, Executive Director Web Survey Design Guidelines Simple text for different browsers – no graphics. –Different browsers render web pages differently. Few questions per screen or simply few questions. Easy to navigate. Short and plain. No scrolling. Clear and encouraging error or warning messages. Every question answered in a similar way - consistent. –Radio buttons, drop downs. ADA compliant. Introduction page or paragraph. Easy to read. –Must see definitions of sponsored research. Can present questions in response to answers. Dillman, D.A (December). Mail and Internet Surveys, The Tailored Design Method. 2nd Ed. New York: John Wiley & Sons.

Duane Webster, Executive Director Library User Survey

Duane Webster, Executive Director Data was collected at seven main campus libraries and seven academic health science libraries in the U.S., between January, 2003 and January, Recent Data Collection Activities Main University LibrariesAcademic Medical Libraries University of ColoradoUniversity of Connecticut Health Center University of ConnecticutUniversity of North Carolina University of Texas Medical Branch Oregon State UniversityUniversity of Texas Southwestern University of Utah University of Virginia Washington University

Duane Webster, Executive Director More than 45,000 networked electronic services uses were surveyed. At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities. MINES is also being conducted by the Ontario Council of University Libraries (OCUL) in in conjunction with ARL. Recent Data Collection Activities

Duane Webster, Executive Director Purpose of Use By Location Main Campus Libraries 2003 – 2005 In the Library n = 7,618 On-Campus, not in the Library n = 6,641 All Usage n = 19,271 72% of sponsored research usage of electronic resources occurred outside the library; 83% of this took place on campus. Off-Campus n = 5,012 26% 3% 5% 66% 33% 2% 6% 59% 24% 2% 11% 63% 14% 2% 21% 63%

Duane Webster, Executive Director Location of Users Medical Library Main Library All Libraries 30% 40% 60% 70% 76% 24% In Library Users Remote Users 76% 24%

Duane Webster, Executive Director Demographics by Location of User Main Libraries On Campus, Not in the Library n = 6,391 Inside the Library n = 7,064 Off-Campus n = 4,953 Total Users n = 18,408 20%8% 27% 45% 27% 39% 12% 22% 40% 3% 33% 24% 38% 62% In the Library Outside the Library Graduate Students Faculty, Staff, Research Fellows Undergraduate Students All Other Users

Duane Webster, Executive Director Electronic Services Sponsored Research Use Compared to Print Journal and Total Library Use Main Libraries E-Resources Print Journals Total Use

Duane Webster, Executive Director Where are the most critical assessment needs and opportunities? Complementing LibQUAL+ ® with additional measures. Developing impact studies on user success, economic value, and community return on investment. Moving target: what is a digital library? E-Resources: understanding usage. Gaining acceptance and use of standard measures for e-resources. Building a climate of assessment throughout library.

Duane Webster, Executive Director What are the lessons learned? Understanding changes in users approach to information resources. Service quality improvement is a key factor. Understanding the impact of e-resources on library services - TRL. Learning how to compete with Google. Upfront investment in design and development. Making the assessment service affordable, practical, & effective. Assessment needs to be satisfying and fun.

Duane Webster, Executive Director In Closing As higher education is challenged on accountability and effectiveness issues so will libraries. A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work. Basic questions of role, vision, and impact must be answered by library community.

Duane Webster, Executive Director Selected References Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, [Available at Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, [Available at Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): [Available at Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, [Available at Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, [Available at Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, [Available at Selected References