Project web site – TM Ottawa, Canada June 14, 2006 Presented by: Dr. Colleen Cook, Dean Texas A&M University Quantitative and Qualitative Methods for Assessing and Improving Outcomes and Service Quality

Why Assess? Why Assess? “ In an age of accountability, there is a pressing need for an effective…process to evaluate and compare research libraries.” u 700 participants in LibQUAL+™ u 123 Association of Research Libraries (ARL) alone, over $3.4 billion dollars were expended in 2003/2004 Note. M. Kyrillidou and M. Young. (2005). ARL Statistics Washington, D.C.: ARL, p.5.

Libraries Remain a Credible Resource in 21 st Century Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment. 98% agree with statement, “My … library contains information from credible and known sources.”

Changing Behaviors Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment. Recent Survey: Only 15.7% agreed with the statement “The Internet has not changed the way I use the library.”

Faculty: Dependence on Electronic Resources Will Increase “I will become increasingly dependent on electronic research resources in the future.”

Research Behavior: Personal Control When searching for print journals for research: Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment. Only 13.9% ask a librarian for assistance Only 13.9% ask a librarian for assistance Only 3.2% consider consulting a librarian a preferred way of identifying information Only 3.2% consider consulting a librarian a preferred way of identifying information

Total Circulation Note. M. Kyrillidou and M. Young. (2005). ARL Statistics Washington, D.C.: ARL, p.6.

Reference Transactions Note. M. Kyrillidou and M. Young. (2005). ARL Statistics Washington, D.C.: ARL, p.6.

Web Usage Total File Requests - UT Austin Libraries

Enter LibQUAL+™ The necessity of assessment Rapid shifts in information-seeking behavior The reallocation of resources from traditional services and functions

The Challenge of Assessment in Libraries Traditional statistics emphasize inputs, expenditures, acquisitions, holdings, etc. Helping funding agencies understand success of investment No demonstrable relationship between expenditures and service quality Lack of metrics describing outcomes: success from the user’s point of view Need to redesign library services to better meet changing patterns of use Building the climate, tools, and skill set for library assessment

ARL New Measures Initiative Collaboration among member leaders with strong interest in this area Specific projects developed with different models for exploration Intent to make resulting tools and methodologies available to full membership and wider community

LibQUAL+™ Goals Improve mechanisms and protocols for evaluating libraries Develop web-based tools for assessing library service quality Identify best practices in providing library service Support libraries seeking to understand changes in user behavior Assist libraries seeking to re-position library services in the new environment

LibQUAL+™ Outcomes Securing information that contributes meaningfully to planning and improvement efforts at a local level Providing analytical frameworks that institutional staff can apply without extensive training or assistance Helping decision-makers understand success of investments Finding useful inter-institutional comparisons

PERCEPTIONS SERVICE “….only customers judge quality; all other judgments are essentially irrelevant” Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press. The LibQUAL+™ Premise

13 Libraries English LibQUAL+™ Version 4000 Respondents QUAL QUAN QUAL QUAN QUAL PURPOSE DATA ANALYSIS PRODUCT/RESULT Describe library environment; build theory of library service quality from user perspective Test LibQUAL+™ instrument Refine theory of service quality Refine LibQUAL+™ instrument Test LibQUAL+™ instrument Refine theory Unstructured interviews at 8 ARL institutions Web-delivered survey Unstructured interviews at Health Sciences and the Smithsonian libraries to survey administrators Web-delivered survey Focus groups Content analysis: (cards & Atlas TI) Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Vignette Re-tooling Iterative Emergent Libraries English, Dutch, Swedish, German LibQUAL+™ Versions 160,000 anticipated respondents LibQUAL+™ Project Case studies 1 Valid LibQUAL+™ protocol Scalable process Enhanced understanding of user-centered views of service quality in the library environment 2 Cultural perspective 3 Refined survey delivery process and theory of service quality 4 Refined LibQUAL+™ instrument 5 Local contextual understanding of LibQUAL+™ survey responses 6

76 Interviews Conducted York University University of Arizona Arizona State University of Connecticut University of Houston University of Kansas University of Minnesota University of Pennsylvania University of Washington Smithsonian Northwestern Medical

LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred

Dimensions of Library Service Quality Empathy Information Control Responsiveness Symbol Utilitarian space Assurance Scope of Content Ease of Navigation Self - Reliance Library as Place Library Service Quality Model 3 Refuge Affect of Service Reliability Convenience Timeliness Equipment

Affect of Service “I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.” Faculty member

Library as Place “One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.” Faculty member

Library as Place “I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.” Faculty member

Information Control “…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.” Faculty member

Information Control “By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.” Graduate student

Multiple Methods of Listening to Customers Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.

LibQUAL+™ Resources An ARL/Texas A&M University joint developmental effort based on SERVQUAL. LibQUAL+™ initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) Initial project established a expert team, re-grounded SERVQUAL concepts, and designed survey methodology Survey conducted at over 500 libraries resulting in a data base of over half a million user responses NSF funded project to refocus LibQUAL+™ on the National Science Digital Library (NSDL)

Participating Libraries World LibQUAL+™ Survey 2005

Rapid Growth in Other Areas Languages  American English  British English  French  Dutch  Swedish In development  Chinese  Greek  Spanish  German Consortia  Each may create 5 local questions to add to their survey Types of Institutions  Academic Health Sciences  Academic Law  Academic Military  College or University  Community College  European Business  Hospital  Public  State Countries  U.S., U.K., Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia

“22 items” items56-items25-items22-items Affect of Service Service Affect ReliabilityLibrary as Place ReliabilityPersonal Control Information Control Provision of Physical Collections Self-Reliance Information Access Access to Information

Survey Instrument

“And a Box” Why the Box is so Important  About 40% of participants provide open- ended comments, and these are linked to demographics and quantitative data.  Users elaborate the details of their concerns.  Users feel the need to be constructive in their criticisms, and offer specific suggestions for action.

Reliability alpha By Language By Language ServiceInfo.Lib as Group n AffectControlPlaceTOTAL American (all)59, British (all) 6, French (all)

Reliability alpha by University Type By University Type ServiceInfo.Lib as Group n AffectControlPlaceTOTAL Comm Colleges 4, yr Not ARL36, yr, ARL14, Acad Health 3,

Validity Correlations Serv_AffInfo_ConLibPlaceTOTALper Serv_Aff Info_Con LibPlace TOTALper ESAT_TOT EOUT_TOT

Understanding Results Understanding LibQUAL+™ Results Measures the distance between minimally acceptable and desired service quality ratings Measures the distance between minimally acceptable and desired service quality ratings Perception ratings ideally fall within the Zone of Tolerance Perception ratings ideally fall within the Zone of Tolerance

Key to Bar Charts

LibQUAL+™ 2004 Summary Colleges or Universities American English (n = 69,449)

Score Norms Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample. LibQUAL+™ norms have been created at both the individual and institutional level

Institutional Norms for Perceived Means on 25 Core Questions Note: Thompson, B. LibQUAL+  Spring 2002 Selected Norms, (2002).

LibQUAL+™ Interactive Institution Statistics

LibQUAL+™ Adequacy Gap The difference between the minimum and perceived score

In Closing LibQUAL+™ Focuses on success from the users point of view (outcomes) Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense Requires limited local survey expertise and resources Analysis available at local and inter-institutional levels Offers many opportunities for using demographics to discern user behaviors

LibQUAL+™ Resources LibQUAL+™ Website: Publications: Events and Training: LibQUAL+™ Bibliography: LibQUAL+™ Procedures Manual:

LibQUAL+™ Contact Information MaShana Davis  Technical Communications Liaison  Richard Groves  Customer Relations Coordinator  Mary Jackson  LibQUAL+™ Services Manager  Martha Kyrillidou  Director, ARL Statistics and Service Quality Programs  woof