1. www.arl.orgAssociation of Research Libraries Measures for Electronic Use: The ARL E-Metrics Project Statistics in Practice - Measuring & Managing IFLA Satellite Conference 13-25 August 2002 Loughborough, UK Julia C. Blixrud, Association of Research Libraries

2. www.arl.orgAssociation of Research Libraries Forces and Challenges Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution Increasing pressure to maximize use of resources through benchmarking resulting in: –Cost savings –Reallocation

3. www.arl.orgAssociation of Research Libraries ARL New Measures Initiative Collaboration among member leaders with strong interests Specific projects developed with different models for exploration Projects self-funded by interested members Intent to make resulting tools and methodologies available to full membership and wider community Continue to collect, but limit/freeze modifications to existing descriptive measures

4. www.arl.orgAssociation of Research Libraries ARL New Measures Projects Demonstration project for service effectiveness measures (LibQUAL+  ) Identification of measures that demonstrate a library’s contribution to student learning outcomes Investigation of role libraries play in support of the research process Development of tools to address cost effectiveness of library operations (staff allocation, ILL/DD study) Project to define usage measures for electronic information resources (E-metrics)

5. www.arl.orgAssociation of Research Libraries Electronic Serials Expenditures

6. www.arl.orgAssociation of Research Libraries Percentages of Acquisitions Dollars Devoted to Electronic Resources

7. www.arl.orgAssociation of Research Libraries Average Yearly Increases in Electronic Resources and Total Library Materials Expenditures

8. www.arl.orgAssociation of Research Libraries Need for Networked Data & Statistics Financial Support –To justify - make a case for continued current support for digital collections –To make a case for additional support for technology & infrastructure Funding

9. www.arl.orgAssociation of Research Libraries Need for Networked Data & Statistics Better Internal Processes –To measure & track changes in internal processes –To enable better decision-making in allocating & prioritizing resources & needs –To enable assessment of service quality in a networked environment Infrastructure

10. www.arl.orgAssociation of Research Libraries Need for Networked Data & Statistics Institutional Comparisons –For benchmarking digital services –To enable competition for resources with other departments on campus For Comparisons

11. www.arl.orgAssociation of Research Libraries Need for Networked Data & Statistics Need for accurate reporting of network use Need for accurate estimates of per client use Ability to compare overlapping coverage Need the ability to pressure vendors to price according to the library’s real need Vendor Negotiation

12. www.arl.orgAssociation of Research Libraries ARL E-Metrics Project Three phases: –Initial Phase (May-October 2000): What do we know? Inventory of current practices at ARL libraries as to statistics, measures, processes, and activities that pertain to networked resources and services. –Second Phase (November 2000-June 2001): What can we collect? Identified and field tested an initial draft set of statistics and measures –Final phase (July 2001-December 2001): What difference does this make? Build linkages to: educational outcomes/impact, research, technical infrastructure

13. www.arl.orgAssociation of Research Libraries Project Participants AlbertaMarylandUSC Arizona StateU. MassTexas A & M AuburnNebraskaVirginia Tech ChicagoNotre DameW. Ontario ConnecticutPennsylvaniaWisconsin CornellPenn StateYale Illinois-ChicagoPittsburghLC ManitobaPurdueNYPL

14. www.arl.orgAssociation of Research Libraries Project Leaders Co-Chairs –Rush Miller, University of Pittsburgh –Sherrie Schmidt, Arizona State University Information Use Management and Policy Institute, Florida State University –Charles R. McClure –Wonsik “Jeff” Shim –John Carlo Bertot ARL –Duane Webster –Martha Kyrillidou

15. www.arl.orgAssociation of Research Libraries Working Definition of Networked Services Those electronic information resources and/or services that users access electronically via a computer network: 1.From on-site in the library 2.Remote to the library – but from a campus facility 3.Remote from the library & campus

16. www.arl.orgAssociation of Research Libraries Networked Information Resources Locally Licensed Databases Regional or Statewide Consortia Licensed Databases Aggregated Databases Publishers Databases Publicly Available (Web) Resources

17. www.arl.orgAssociation of Research Libraries Networked Information Services Access to text & numerical databases, electronic journals, electronic books, e-lists, e-mail Instruction, training & workshops Reference & information services Virtual reference Interlibrary loan, document delivery IT infrastructure Institutional & personal portals

18. www.arl.orgAssociation of Research Libraries Vendor Statistics Working Group 12 major ARL vendors met with project team in Denver preceding 2000 ACRL Meeting Academic Press/IDEAL * Elsevier/Science Direct Lexis/Nexis Ovid Bell & Howell Gale Group ISI * † netLibrary Silver Platter * EBSCO JSTOR OCLC/First Search * Unable to attend Denver Meeting † Nonparticipant in project.

19. www.arl.orgAssociation of Research Libraries Related E-Metrics Projects–Some Have Formal Collaboration with ARL ICOLC Guidelines Equinox (European Union) ISO Library Statistics standard under development NISO – updating Z39.7 standard on statistics NCLIS – working on standardized online database usage statistics IMLS – Sponsored research to develop statistics for public libraries CLIR – Consultants to review this area DLF – Measuring digital library use PALS (Publishers and Libraries Solutions Committee) represents the scholarly publishers and academic libraries in the UK CENDI Web Metrics

20. www.arl.orgAssociation of Research Libraries Organizational Context Technical Infrastructure Information Content Information Services SupportManagement (D3) Cost of digital collection and management (R1) # of e-full- text journals (R2) # of e- reference sources (R3) # of e-books (D1) Size of digital library collection (U5) # virtual visits to the resources (P2) % of virtual visits of all visits (P3) % of e-books to all books (D2) Use of digital collection (U1) # e-ref transactions (U2) # of logins (sessions) to e- databases (U3) # of queries in e-databases (U4) Items requested in e- databases (U5) # virtual visits (P1) % of e-ref of total ref (D3) Cost of digital collection construction & management (C1) Cost of e-full-text journals (C2) Cost of e-ref sources (C3) cost of e-books (C4) Library exp. for bibl-utilities, networks & consortia (C5) External exp. for bibl-utilities, networks & consortia (D3) Cost of digital collection construction.. Library Networked Services and Resources

21. www.arl.orgAssociation of Research Libraries Recommended Statistics & Measures Patron Accessible Electronic Resources (R1-3) Use of Networked Resources & Services (U1-5) Expenditures for Networked Resources & Related Infrastructure (C1-3) Library Digitization Activities (D1-3) Performance Measures (P1-3)

22. www.arl.orgAssociation of Research Libraries Patron Accessible Electronic Resources R1 – Number of electronic full-text journals R2 – Number of electronic reference sources R3 – Number of electronic books

23. www.arl.orgAssociation of Research Libraries Use of Networked Resources & Related Infrastructure U1 – Number of electronic reference transactions U2 – Number of logins (sessions) to electronic databases U3 – Number of queries (searches) in electronic databases U4 – Items requested in electronic databases U5 – Virtual visits to library’s website and catalog

24. www.arl.orgAssociation of Research Libraries Expenditures for Networked Resources & Related Infrastructure C1  Cost of electronic full-text journals C2  Cost of electronic reference sources C3  Cost of electronic books C4  Library expenditures for bibliographic utilities, networks & consortia C5  External expenditures for bibliographic utilities, networks & consortia

25. www.arl.orgAssociation of Research Libraries Library Digitization Activities D1 – Size of library digital collection D2 – Use of library digital collection D3 – Cost of digital collection construction & management (Collecting these data requires staff familiar with the digital environment.)

26. www.arl.orgAssociation of Research Libraries Characteristics of Each Recommended Measure Definition Rationale Unit of Measure Data source Frequency Process Related Issues

27. www.arl.orgAssociation of Research Libraries R1 – Number of Electronic Full-text Journals Definition - Number of electronic full-text journal subscriptions – by individual institution or consortia licensing. Rationale – Documents degree of expansion of electronic subscriptions available – can be used to show good coverage & need for more funding. Unit of measure – the journal subscription Data source – local or vendors. Frequency – annual, monthly, etc. Process – parse into database or spreadsheet, update dynamically from local catalog or vendor record.

28. www.arl.orgAssociation of Research Libraries U1 – Number of Electronic Reference Transactions Definition - number of electronic reference transactions – via e- mail, WWW form, etc. Rationale – libraries are interested in tracking the development of new electronic services. Attempt to measure reference transactions through new electronic tools and services. Unit of Measure – request count, time it took. Data Source – local server, manual tally, e-mail count. Frequency – daily, monthly, annually, etc. Process – clarify process, identify activity points, identify collectors of data, consolidate data. Related Issues –This measure may have to broken down into additional data types – time, type of query, type of interaction, scheduling issues, measures of quality and reliability.

29. www.arl.orgAssociation of Research Libraries Performance Measures P1 – Percentage of electronic reference transactions of total reference P2 – Percentage of virtual visits of all library visits P3 – Percentage of electronic books to all monographs Percentage of electronic journals to serial subscriptions

30. www.arl.orgAssociation of Research Libraries E-Metrics Assessment Plan Have administrative clarity Reflect institutional structure and staff functions Balance stakeholder needs with availability of data Provide for the input, structure, housing & archiving of data Propose a structure to disseminate data, reports & information – dynamic web intranet (portal)

31. www.arl.orgAssociation of Research Libraries E-Metrics Implementation Process 1.Preparation Immediate Long term 2.Identification of Tasks, Data and Needs 3.Data Collection 4.Information Management 5.Reporting, Dissemination & Feedback

32. www.arl.orgAssociation of Research Libraries E-Metrics Next Steps Call for participation among members to test proposed measures for 2002/2003 (over 35 participants to date) –Examination of the deliverables from the first phases –Collect FY02 totals Compilation Data analysis Distribution for discussion –Analysis of approaches –Best practices for work processes From E-metrics project Locally developed

33. www.arl.orgAssociation of Research Libraries E-Metrics Next Steps Continued work with vendors through international COUNTER project Continued work with international standards activities Workshops and training to develop necessary data analysis skills

34. www.arl.orgAssociation of Research Libraries (Carla Stoffle, University of Arizona) Developing measures and evaluation techniques for networked services will take time, effort, and on-going learning on everyone’s part – but we must begin now. We not only need to measure things in new ways but we also need to measure new things. (Sherrie Schmidt, Arizona State University)

35. www.arl.orgAssociation of Research Libraries Project Documents Measures for Electronic Resources (E-Metrics) Part 1: Project Background and Phase One Report Part 2: Phase Two Report Part 3: E-Metrics Instructional Module Part 4: Data Collection Manual Part 5: Library and Institutional Outcomes www.arl.org/stats/newmeas/emetrics/

36. www.arl.orgAssociation of Research Libraries Julia C. Blixrud Director of Information Services Association of Research Libraries 21 Dupont Circle, Ste 800 Washington, DC 20036 jblix@arl.org 202-296-2296 ext. 133 202-872-0884 (fax) 202-251-4678 (cell)