Making Sense of Usage Statistics for Online Databases Challenges, Lessons, and Strategies in a Statewide Context Coalition for Networked Information Task Force Meeting, April 16, 2004, Washington, DC William E. Moen School of Library and Information Sciences Texas Center for Digital Knowledge University of North Texas Denton, TX Charles R. McClure School of Library and Information Studies Information Institute Florida State University John Carlo Bertot Tallahassee, FL 32306

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 2 Overview Why and what of usage statistics Library of Texas/TexShare experience Data issues The case of metasearch Interpretation issues Management issues Final comments

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 3 Why care about usage statistics Major investment by libraries and consortia in licensed resources Usage information as basis for: Securing funding Allocating funding Choosing or deselecting resources Understanding patterns of use A non-intrusive method for viewing users behaviors Log analysis for user behaviors Metasearch applications change nature of the usage and utilization of databases

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 4 What statistics to report At an abstract level: To what extent do users engage with a resource? Sessions Searches To what extent do users utilize a resource Viewing result lists Viewing full records of particular results Downloading or viewing full text (if applicable) Downloading data for further processing (if applicable)

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 5 The issues Do database vendors provide usable usage data for libraries Do database vendors provide comparable data How do libraries deal with heterogeneous data How to automate analysis and reporting of heterogeneous usage data How can data be best integrated into library decision making

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 6 TexShare A cooperative program to improve service to Texans. It maximizes the effectiveness of library expenditures by enabling libraries to: Share staff expertise Share library resources in print and electronic formats Pursue joint purchasing agreements for information services Encourage cooperative development and deployment of information resources and technologies The TexShare database service Statewide licensing of databases for users of academic, public libraries, and libraries of clinical medicine Core databases TexSelect databases

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 7 Library of Texas Envisioned as a service-based virtual library that will enable Texans to search an extensive array of resources The LOT initiative covers four basic components: Indexing and preserving electronic government documents Providing a statewide resource discovery service Training librarians on electronic resources Continuing to offer a wide selection of TexShare databases

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 8 Longitudinal usage analysis Part of an evaluation project for LOT Detailed analysis and reanalysis of vendor supplied data (FY01-FY03) Purposes: Compile database usage and trends Determine the extent to which analysis could be automated

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 9 The TexShare statistics challenge Different vendors Multiple database per vendors Changing number of databases per reporting period Creating comparable usage data Sessions Searches Documents: Number of full-text downloads Verification of historical data

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 10 TexShare databases available VendorFY01FY02FY03 BIG CHALK11 EBSCO GALE12 10 GROLIER44 HANDBOOK OF TEXAS111 NETLIBRARY111 OCLC1110 PROQUEST552 R.R. BOWKER22 TETON DATA SYSTEM111 TDNET11 Total636547

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 11 Heterogeneous vendor data EBSCO Logins Searches Abstract Full Text Articles  Transformation: Logins  Sessions Searches  Searches Full Text Articles  Documents Gale Retrievals Searches Turnaways Views Sessions Total connect time  Transformation: Sessions  Sessions Searches  Searches Views  Documents

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 12 Heterogeneous vendor data Proquest Total visitors Total pageviews Total hits Total bytes transferred Average visitors per day Average pageviews per day Average hits per day Average bytes transferred per day Average pageviews per visitor Average hits per visitor Average bytes per visitor Average length of sessions  Transformation Total Visitors  Sessions “tx” directory tree Page Views  * Searches “index.html” directory Page Views  **Documents * Searches = number of “tx” Page Views / 4 ** Documents = number of “index.html” Page Views / 5

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 13 Statistics available per vendor SessionsSearchesDocumentsLib. Type BIG CHALKYES EBSCOYES GALEYES GROLIERYESNOYESNO NETLIBRARYNO YES OCLCYES YES * PROQUESTYES NO R.R. BOWKERYES TETON DATA SYSTEMYES NO * Statistics provided for public libraries only

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 14 Summary statistics AcademicPublicUndifferentiated Sessions4,896,326922,041111,251 Searches13,398,5062,922,27715,099* Documents8,864,8931,441,872373,371 AcademicPublicUndifferentiated Sessions6,125,2151,549,281269,143 Searches16,932,8243,317,900179,005* Documents9,483,0051,382,2231,421,540 FY2002 FY2003 * One vendor does not provide search statistics

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 15 Summary stats per vendor Vendor (databases)SessionsSearches EBSCO (27)4,417,4299,859,812 GALE (12)1,418,6516,240,454 GROLIER (4)69,328* NETLIBRARY (1)** OCLC (10)1,268,6673,164,049 PROQUEST (5)616,636657,486 R.R. BOWKER (2)112,366435,339 FY2003 * Grolier doesn’t provide data for searches ** NetLibrary doesn’t provide data for sessions and searches

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 16 Data issues Difficulties in obtaining complete data from vendors Differences among vendors in their definitions (or lack thereof) of key terms such as sessions, downloads, etc. Problems encountered by TSLAC in maintaining and collating accurate data from the various vendors Problems encountered when retrieving vendor data from earlier months and having that data differ from the data originally retrieved from the vendor site Difficulties in developing and integrating standardized reporting tools across different vendor data to summarize usage across the TexShare databases. Difficulties in utilizing NISO Library Statistics Standard

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 17 Data issues BOTTOM LINE: Better to have limited data and use them wisely than to have no data at all! As research and development continues we will have higher quality data and better methods to insure normalization across vendors.

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 18 Emerging community agreements Working towards vendor data reporting standards Standards efforts NISO Z39.7 Library Statistics standard ISO 2789 Library Statistics standard Both focus on e-metric (data element) definitional standards Library/consortia efforts ICOLC (International Coalition of Library Consortia) Guidelines for data elements and reporting Publisher/vendor efforts Project COUNTER (Counting Online Usage of Networked Electronic Services) Audited data reporting of vendor usage statistics in standard format Code of Practice to which vendors adhere and adherence is independently verified

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 19 Vendor usage E-metrics From NISO Rejected Sessions (turnaways) Commercial Services Sessions Commercial Services Searches (queries) Commercial Services Full-Content Units Examined Commercial services descriptive records examined

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 20 Usage data in metasearch Metasearch Single search interface to multiple resources Library of Texas Resource Discovery Service Opportunity to define and control usage statistics in the metasearch application, e.g.:  Total Sessions  Total Searches  Total Searches per Target Resource  Total Full Text Downloads  Total Link Outs to Target Native Interfaces per Target Resource  Total Get It Requests  Total ILL Submitted Interaction with metasearch and vendor statistics What are user behaviors in the metasearch environment?

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 21 LOT Search Interface

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 22 RDS Analyzer

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 23 Interpretation issues What constitutes significant use? What measures point to impact? Approaches: Annual usage data could compare the statistics per year by library (1300 academic and public libraries) Annual use data could compare the statistics per year per person (approximately 21.3 million residents) Cost of the TexShare databases for each fiscal year could be used to determine the average cost per session (or download) for (1) all libraries or by academic or public library, and (2) on a per capita basis for residents of Texas Assuming the ability to provide a “legal service population” for each academic and public library participating in TexShare, the use data can also be normalized on a per capita basis for each library – which could indicate possible variations in the use (and therefore the cost) for each library Cost avoidance without regard to usage

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 24 Management issues Organizing for data management: Who has overall responsibility for data management and who has specific responsibility for  Collecting the data  Designing and maintaining a management information system  Entering the data in a management information system  Verifying data with vendors  Analysis of data  Reporting data

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 25 Management issues (Con’t) What is the budget available to support data management of online usage statistics? Who are the audiences that the usage statistics will be reported to? Do staff have the necessary skills to collect, analyze, and report this data? Are there analyses of usage data that should remain internal to the organization?

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 26 Parting Shots Online database usage statistics are essential for ongoing evaluation of online database services Basic usage statistics can be used to develop a range of performance measures and indicators for outcomes assessment Vendors must pay attention to standards compliance in usage stats The usage statistics can be used to justify or refine database purchases

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 27 YOU CAN DO THIS! As libraries, consortia, and statewide digital libraries continue to grow and expand they will continue to rely on the provision of online databases. Ongoing collection, analysis, and reporting of usage statistics is an essential component for collection development as well as overall library services planning.

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 28 E-Metrics Instructional System

Moen, McClure, Bertot CNI Task Force Meeting--April 16, Washington, DC 29 References The Library of Texas  Library of Texas Resource Discovery Service  ZLOT Project  NISO Z39.7 Library Statistics standard  Project Counter  ICOLC (International Coalition of Library Consortia)  E-Metrics Instructional System 