of Research Libraries Library Metrics in a Changing Environment National Seminar of Libraries in Malaysia, May 25-27, 2004 Bruce Thompson, Texas A&M University Julia C. Blixrud, Association of Research Libraries Colleen Cook, Texas A&M University Martha Kyrillidou, Association of Research Libraries

of Research Libraries Forces and Challenges Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution Increasing pressure to maximize use of resources through benchmarking resulting in: –Cost savings –Reallocation

of Research Libraries “ In an age of accountability, there is a pressing need for an effective and practical process to evaluate and compare research libraries. In the aggregate, among the 124 Association of Research Libraries (ARL) alone, over $3.2 billion dollars were expended in 2000/2001 to satisfy the library and information needs of the research constituencies in North America. ” The Imperative for our Research Note. M. Kyrillidou and M. Young. (2002). ARL Statistics Washington, D.C.: ARL, p.5.

of Research Libraries Note. M. Kyrillidou and M. Young. (2002). ARL Statistics Washington, D.C.: ARL, p.7. Total Circulation

of Research Libraries Note. M. Kyrillidou and M. Young. (2002). ARL Statistics Washington, D.C.: ARL, p.7. Reference Transactions

of Research Libraries ARL New Measures Initiative Collaboration among member leaders with strong interests Specific projects developed with different models for exploration Projects self-funded by interested members Intent to make resulting tools and methodologies available to full membership and wider community

of Research Libraries ARL New Measures Projects Project to define usage measures for electronic information resources (E-metrics/COUNTER Online Metrics) NSF NSDL grant to identify the dimensions of digital library service quality (e-QUAL or “digiqual”) Survey on User Demographics and Purpose of Use for Electronic Resources (Project MINES) Measuring Library Service Quality (LibQUAL+  ) Identification of measures that demonstrate a library’s contribution to student learning outcomes Investigation of role libraries play in support of the research process Development of tools to address cost effectiveness of library operations (staff allocation, ILL/DD study)

of Research Libraries ARL Data Expenditures for electronic resources account for 25%, on average, of ARL institutions’ library materials budgets. ARL libraries reported spending more than $228 million on electronic resources. ARL libraries reported a total of $21,470,716 in additional funds spent on their behalf through a centrally funded consortium for purchasing electronic products and services. Expenditures for electronic serials have increased by 171% since the survey, and by more than 1800% since they were first reported, in (see graph, below). Source: Mark Young and Martha Kyrillidou, ARL Supplementatry Statistics (Washington, DC: Association of Research Libraries, 2004)

of Research Libraries Mark Young and Martha Kyrillidou, ARL Supplementary Statistics (Washington, DC: Association of Research Libraries, 2004)

of Research Libraries Mark Young and Martha Kyrillidou, ARL Supplementary Statistics (Washington, DC: Association of Research Libraries, 2004)

of Research Libraries Need for Networked Data & Statistics Financial Support –To justify - make a case for continued current support for digital collections –To make a case for additional support for technology & infrastructure Funding

of Research Libraries Need for Networked Data & Statistics Better Internal Processes –To measure & track changes in internal processes –To enable better decision-making in allocating & prioritizing resources & needs –To enable assessment of service quality in a networked environment Infrastructure

of Research Libraries Need for Networked Data & Statistics Institutional Comparisons –For benchmarking digital services –To enable competition for resources with other departments on campus For Comparisons

of Research Libraries Need for Networked Data & Statistics Need for accurate reporting of network use Need for accurate estimates of per client use Ability to compare overlapping coverage Need the ability to pressure vendors to price according to the library’s real need Vendor Negotiation

of Research Libraries ARL E-Metrics Project Three phases: –Initial Phase (May-October 2000): What do we know? Inventory of current practices at ARL libraries as to statistics, measures, processes, and activities that pertain to networked resources and services. –Second Phase (November 2000-June 2001): What can we collect? Identified and field tested an initial draft set of statistics and measures –Final phase (July 2001-December 2001): What difference does this make? Build linkages to: educational outcomes/impact, research, technical infrastructure

of Research Libraries ARL E-Metrics Project Number of libraries collecting e-metrics data elements increased over a period of three years: 25 libraries in libraries in libraries in 2004 Data elements will be part of the annual ARL Supplementary Survey in

of Research Libraries Data Elements: Patron Accessible Electronic Resources R1 – Number of electronic full-text journals R2 – Number of electronic reference sources R3 – Number of electronic books

of Research Libraries Data Elements: Use of Networked Resources & Related Infrastructure U1 – Number of electronic reference transactions U2 – Number of logins (sessions) to electronic databases U3 – Number of queries (searches) in electronic databases U4 – Items requested in electronic databases U5 – Virtual visits to library’s website and catalog

of Research Libraries Data Elements: Expenditures for Networked Resources & Related Infrastructure C1  Cost of electronic full-text journals C2  Cost of electronic reference sources C3  Cost of electronic books C4  Library expenditures for bibliographic utilities, networks & consortia C5  External expenditures for bibliographic utilities, networks & consortia

of Research Libraries Data Elements: Library Digitization Activities D1 – Size of library digital collection D2 – Use of library digital collection D3 – Cost of digital collection construction & management (Collecting these data requires staff familiar with the digital environment.)

of Research Libraries E-Metrics Next Steps Continued work with vendors through international Project COUNTER Continued work with national and international standards activities Workshops and training to develop necessary data analysis skills

of Research Libraries COUNTER Online Metrics ARL was a founding members of COUNTER and COUNTER goals include: developing, reviewing, disseminating and gaining support for an internationally agreed Code of Practice governing the recording and exchange of online usage data and other appropriate Codes of Practice relating to online publications; developing an organisational framework for implementation of and compliance with such Codes of Practice; contributing to the public, commercial and professional understanding of online information use.

of Research Libraries 13 Libraries English LibQUAL+™ Version 4000 Respondents QUAL QUAN QUAL QUAN QUAL PURPOSE DATA ANALYSIS PRODUCT/RESULT Describe library environment; build theory of library service quality from user perspective Test LibQUAL+™ instrument Refine theory of service quality Refine LibQUAL+™ instrument Test LibQUAL+™ instrument Refine theory Unstructured interviews at 8 ARL institutions Web-delivered survey Unstructured interviews at Health Sciences and the Smithsonian libraries to survey administrators Web-delivered survey Focus groups Content analysis: (cards & Atlas TI) Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis Vignette Re-tooling Iterative Emergent Libraries English, Dutch, Swedish, German LibQUAL+™ Versions 160,000 anticipated respondents LibQUAL+™ Project Case studies 1 Valid LibQUAL+™ protocol Scalable process Enhanced understanding of user-centered views of service quality in the library environment 2 Cultural perspective 3 Refined survey delivery process and theory of service quality 4 Refined LibQUAL+™ instrument 5 Local contextual understanding of LibQUAL+™ survey responses 6

of Research Libraries LibQUAL+  Brief History Experience with SERVQUAL in many libraries over the last 10 years Texas A&M SERVQUAL assessment Meeting of interested ARL libraries (ALA Midwinter 2000) Pilot with 12 ARL libraries (spring 2000) – 5,000 responses External funding through FIPSE, U.S. Department of Education (September 2000) 43 libraries participated spring 2001 – 20,000 responses 164 libraries participated spring 2002 – 78,000+ responses 308 libraries participated spring 2003 – 125,000+ responses 208 libraries participating spring 2004 – 110,000+ responses Consortial and related associations interest International interest NSF NSDL funding to develop an understanding of service quality in the digital library environment (e-QUAL or ‘digiqual’)

of Research Libraries Dimensions items56-items25-items22-items Affect of Service Service Affect ReliabilityLibrary as Place ReliabilityPersonal Control Information Control Provision of Physical Collections Self-RelianceInformation Access Access to Information

of Research Libraries LibQUAL+ TM Participants Spring 2000 Year 0 13 Year 1 42 Year Year 3 Spring 2001 Spring 2002 Spring Year 4 Spring 2004

of Research Libraries Mean Perceived Scores 2001/2002 Trend (n=34)

of Research Libraries Contributions of LibQUAL+™ Web-based instrument makes little demand of local resources while compiling robust dataset Grounded questions yield data of sufficient granularity to be of local use Normative data across cohort groups Surfaces “Best Practices”

of Research Libraries Library Values Library values are reflected in: physical environment (Library as Space) warmth, empathy, reliability and assurance of library staff (Affect of Service) ability to control the information universe in an efficient way (Information Control) and are unifying and powerful forces for: Overcoming language and cultural barriers Bridging the worlds of our users Improving library services Advancing the betterment of individuals and societies

of Research Libraries LibQUAL+  Related Documents LibQUAL+  LibQUAL+  Web Site LibQUAL+  LibQUAL+  Bibliography Survey Participants Procedures Manual

of Research Libraries (Carla Stoffle, University of Arizona) Developing measures and evaluation techniques for networked services will take time, effort, and on-going learning on everyone’s part – but we must begin now. We not only need to measure things in new ways but we also need to measure new things. (Sherrie Schmidt, Arizona State University)