of Research Libraries Next Generation Statistics for Libraries: CAUL Forum 20 January 2003 Sydney, NSW The ARL E-Metrics Project Mary E. Jackson Senior Program Officer for Access Services Association of Research Libraries

of Research Libraries ARL New Measures Initiative Collaboration among member leaders with strong interests Specific projects explore different models Self-funded projects

of Research Libraries ARL New Measures Initiative Make resulting tools and methodologies available to full membership and wider community Continue to collect, but limit/freeze modifications to existing descriptive measures

of Research Libraries ARL New Measures Projects Demonstration project for service effectiveness measures –LibQUAL+  Identification of measures that demonstrate a library’s contribution to student learning outcomes Investigation of role libraries play in support of the research process

of Research Libraries ARL New Measures Projects Development of tools to address cost effectiveness of library operations –staff allocation, Assessing ILL/DD Services study Project to define usage measures for electronic information resources –E-metrics Project

of Research Libraries Electronic Serials Expenditures

of Research Libraries Percentages of Acquisitions Dollars Devoted to Electronic Resources

of Research Libraries Average Yearly Increases in Electronic Resources and Total Library Materials Expenditures

of Research Libraries Project Leaders Co-Chairs –Rush Miller, University of Pittsburgh, and Sherrie Schmidt, Arizona State University Information Use Management and Policy Institute, Florida State University –Charles R. McClure, Wonsik “Jeff” Shim, and John Carlo Bertot ARL –Duane Webster and Martha Kyrillidou

of Research Libraries Project Participants Alberta Arizona State Auburn Chicago Connecticut Cornell Illinois - Chicago Library of Congress Manitoba Maryland Massachusetts Nebraska New York Public Notre Dame Pennsylvania Penn State Pittsburgh Purdue Southern California Texas A & M Virginia Tech Western Ontario Wisconsin Yale

of Research Libraries Vendor Statistics Working Group 12 major ARL vendors met with project team in Denver prior to 2000 ACRL Meeting Academic Press/IDEAL * Elsevier/Science Direct Lexis/Nexis Ovid Bell & Howell Gale Group ISI * † netLibrary Silver Platter * EBSCO JSTOR OCLC/First Search * Unable to attend Denver Meeting † Nonparticipant in project.

of Research Libraries Working Definition of Networked Services Electronic information resources and/or services that users access electronically via a computer network: 1.From on-site in the library 2.Remote to the library – but from a campus facility 3.Remote from the library & campus

of Research Libraries Networked Information Resources Locally Licensed Databases Regional or Statewide Consortia Licensed Databases Aggregated Databases Publishers Databases Publicly Available (Web) Resources

of Research Libraries ARL E-Metrics Project: Phase 1 May-October 2000 What do we know? Inventory of current practices at ARL libraries of statistics, measures, processes, and activities that pertain to networked resources and services.

of Research Libraries ARL E-Metrics Project: Phase 2 November 2000-June 2001 What can we collect? Identified and field tested an initial draft set of statistics and measures.

of Research Libraries ARL E-Metrics Project: Phase 3 July 2001-December 2001 What difference does this make? Build linkages to: educational outcomes/impact, research, technical infrastructure.

of Research Libraries Recommended Statistics & Measures Patron Accessible Electronic Resources (R1-3) Use of Networked Resources & Services (U1-5) Expenditures for Networked Resources & Related Infrastructure (C1-3) Library Digitization Activities (D1-3) Performance Measures (P1-3)

of Research Libraries Patron Accessible Electronic Resources R1 – Number of electronic full-text journals R2 – Number of electronic reference sources R3 – Number of electronic books

of Research Libraries Use of Networked Resources & Related Infrastructure U1 – Number of electronic reference transactions U2 – Number of logins (sessions) to electronic databases (db) U3 – Number of queries (searches) in electronic db U4 – Items requested in electronic db U5 – Virtual visits to library’s website and catalog

of Research Libraries Expenditures for Networked Resources & Related Infrastructure C1  Cost of electronic full-text journals C2  Cost of electronic reference sources C3  Cost of electronic books C4  Library expenditures for bibliographic utilities, networks, & consortia C5  External expenditures for bibliographic utilities, networks, & consortia

of Research Libraries Library Digitization Activities D1 – Size of library digital collection D2 – Use of library digital collection D3 – Cost of digital collection construction & management (Collecting these data requires staff familiar with the digital environment.)

of Research Libraries Performance Measures P1 – Percentage of electronic reference transactions of total reference P2 – Percentage of virtual visits of all library visits P3 – Percentage of electronic books to all monographs

of Research Libraries Characteristics of Each Recommended Measure Definition Rationale Unit of Measure Data source Frequency Process Related Issues

of Research Libraries R1 – Number of Electronic Full-text Journals Definition - Number of electronic full-text journal subscriptions – by individual institution or consortia licensing. Rationale – Documents degree of expansion of electronic subscriptions available – can be used to show good coverage & need for more funding. Unit of measure – the journal subscription

of Research Libraries R1 – Number of Electronic Full-text Journals Data source – local or vendors. Frequency – annual, monthly, etc. Process – parse into database or spreadsheet, update dynamically from local catalog or vendor record.

of Research Libraries U1 – Number of Electronic Reference Transactions Definition - number of electronic reference transactions – via , WWW form, etc. Rationale – libraries are interested in tracking the development of new electronic services. Attempt to measure reference transactions through new electronic tools and services. Unit of Measure – request count, time it took.

of Research Libraries U1 – Number of Electronic Reference Transactions Data Source – local server, manual tally, e- mail count. Frequency – daily, monthly, annually, etc. Process – clarify process, identify activity points, identify collectors of data, consolidate data. Related Issues –This measure may have to broken down into additional data types – time, type of query, type of interaction, scheduling issues, measures of quality and reliability.

of Research Libraries Outcomes of the Investigation Mixed results Participants wanted more definitive answers Confirmed how difficult, but not impossible, to come to consensus Realized how dependent libraries are for vendors to supply data Recognized need for ongoing effort Reaffirmed importance of using data for decisions

of Research Libraries E-Metrics Next Steps Call for participation among members to test proposed measures for 2002/2003 (over 35 participants to date) –Examination of the deliverables from the first phases –Collect FY02 totals Compilation Data analysis Distribution for discussion –Analysis of approaches –Best practices for work processes From E-metrics project Locally developed

of Research Libraries E-Metrics Next Steps Continued work with vendors through international COUNTER project Continued work with international standards activities Workshops and training to develop necessary data analysis skills

of Research Libraries Project Documents Measures for Electronic Resources (E- Metrics) Part 1: Project Background and Phase One Report Part 2: Phase Two Report Part 3: E-Metrics Instructional Module Part 4: Data Collection Manual Part 5: Library and Institutional Outcomes

of Research Libraries Additional Questions? Please contact: Martha Kyrillidou Senior Program Officer for Statistics and Measurement Association of Research Libraries