Assess for Success: Proving Library Value Steve Hiller Director, Planning and Assessment University of Washington Libraries

Assessment Basics What do we need to know about our communities and customers to make them successful? How do we measure the effectiveness of our services, programs and resources? What do our stakeholders need to know in order to provide the resources needed for a successful library?

Assessment Is More than Numbers Library assessment is a structured process: To learn about our communities To respond to the needs of our users To improve our programs and services To contribute to the success of our communities

Assessment Process Focuses on understanding customer needs and offering services that meet those needs Collects, analyzes and uses quantitative and qualitative data for more effective management and decision-making Emphasizes ongoing communication with customers and stakeholders, Seeks opportunities for collaboration and comparisons within the organization, institution and beyond

What Do We Need to Know About Our Customers? Who are our customers (and potential customers)? What areas/fields/courses are they working in? How do they work? What’s important to them? How do they find information needed for their work? How do they use our services? What would they change? How do they differ from each other in use/needs? Outcomes How do we add value to their work? How do we contribute to their success?

The Value of Community Assessment Identify actual and potential customers Understand needs and use preferences Use funding and staff effectively Understand and address “competition” Encourage community involvement and “ownership” Target marketing, market penetration and outreach Measure, demonstrate, present the value of the library to the community and stakeholders

Thinking Strategically About Library Futures What is our central work and how can we do more, differently, and at less cost? What important services do we provide that others can’t? What advantages do we possess? How is customer behavior changing? How do we add value to our customers work? What are the essential factors responsible for library success now and in the future?

The Challenge for Libraries Traditional statistics and data are no longer sufficient Emphasize inputs – how big and how many Do not tell the library’s story May not align with organizational goals and plans Do not measure service quality Need data from the user’s perspective Need to provide organizational accountability to funders and other stakeholders Need to define where we want to be in 3/5/more years Need to answer: What value do we provide our primary community and how do we show it?

Defining Success in a Digital Environment Crafting new indicators and measures of success Moving from measuring inputs to outputs and outcomes Understanding impact of library/partner roles and services on the community Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing community goals. Reallocating resources and managing capabilities to achieve success. Demonstrating value of library contributions

Some New Metrics for Academic Libraries Uniqueness of collections Value of consortia Administrative and budgetary efficiencies Student outcomes/student learning/graduate success Contributions to faculty productivity Social frameworks/intellectual networks Generating new knowledge Creating the collective good with reusable assets such as an institutional repository or e-science

Good Assessment Starts Before You Begin . . . Some Questions to Ask Define the question What do you need to know and why How will you use the information/results Where/how will you get the information What are appropriate methods Is there existing information New data (where or who will you get it from) Is it cost and resource effective How will you analyze the information Who will act upon the findings

How Do We Get Customer Information? Statistics/data mining (local, institutional) Surveys Focus groups Observation Usability Interviews Embedding Logged activities Comments, suggestions, “over the counter”, being there

Another View of Metrics

Getting The Right Data Isn’t Enough “…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.” Carl Becker Annual Address of the President of the American Historical Association, 1931 Understanding what the data means and how to follow-up are crucial

Make Data Meaningful Summarize Compare Analyze Present Go below the surface to examine results by: Demographic group Users and non-users Stakeholders vs non-stakeholders Compare current data with past; look for trends How can we use the data for action?

Data Presentation: 3 Key Questions What’s the message? Fewer “messages” means greater impact Who’s the audience? Multiple audiences may need different presentations How do we present? Quantitative data - Be graphic , provide meaning/understanding Qualitative data - Be selective, provide corroborating data Keep it simple AND focused!

And Remember the Content!

Moving from Analysis to Action From all of the data, determine what can and should be addressed Prioritize the important/critical action items Make evidence-based decisions Align with mission, vision and goals of parent organization Address what is important to customers Establish action guidelines Who, how, when

What Makes it Hard to Be Evidence-Based What Makes it Hard to Be Evidence-Based? (From Pfeffer and Sutton, 2006) There’s too much evidence There’s not enough good evidence The evidence doesn’t quite apply People are trying to mislead you You are trying to mislead you The side effects outweigh the cure Stories are more persuasive anyways

When the Evidence Isn’t Used

Collection Assessment: The 4 Questions What’s important from the: Library perspective? Customer perspective? Stakeholder perspective? Vendor/publisher perspective(s)?

Some Good Reasons to Assess Collections Meeting user needs Costs/Budget/Space Changes in the information discovery/use processes Accountability to stakeholders, funding agencies Breadth and depth; comprehensiveness Environmental changes (social, information, technical) Changes in publisher/vendor marketing and packaging Return on Investment (ROI); contingent valuation

What Numerical/Cost Data Do We Need? Costs Collections (annual/over time) Titles Packages, consortia Operational Staff Facilities Vendors Usage Print (on-site, check-out, ILL) Electronic/online Type of use Packages, consortial, doc delivery Change over time Cost per use What type of use Downloads Searches Loans Page views Packages, individual titles Global information Collection comparisons Publishing output Bibliometric tools, citation analyses, collection Community demographics

So We Need Systems/Products That: Enable us to track and validate usage Enable us to track item level and package level costs Provide comparative information Provide “canned”/custom reports easily Provide data in flexible formats we can manipulate/analyse Provide information that meets established standards Present snapshot and trend data Are cost-effective

E-Metrics Tools: COUNTER, SUSHI and ERMI Counting Online Usage of NeTworked Electronic Resources Standardized way to count usage Defined reports for journals, databases, and eBooks Standardized Usage Statistics Harvesting Initiative Model for automation of statistics harvesting SUSHI feed: usage data that magically appears! Electronic Resources Management Initiative Can accept SUSHI feeds Easy to generate COUNTER-style reports within ERM module Brings in data from order records to calculate cost per use

Genesis Statistics Detail Title 2007 CPU 2007 Use 2008 CPU 2008 Use Genesis $3.00 649 $3.67 93 2007 payment / 2007 Use = 2007 CPU $1950.71 / 649 = $3.00

Don’t Forget the Qualitative Discovery pathways and user expertise Usability User needs User outcomes What did the information enable the user to accomplish?

Integrated Collection Assessment Collections-centered Size, growth, formats, authors, subjects, comparisons Cost-centered Expenditures, price, inflation, cost per use, return on investment Use-centered Loans, e-usage, purchase requests, ILL Customer centered (secondary) Community demographics, programs, research, bibliometric studies, consortial Customer centered (primary) Behavior, impact, outcomes, success, discovery

What Will We Do With the Information? Analyze and understand Present and share Make good decisions and choices Ensure we add value to the work of our community Customers and stakeholders make the ultimate determination of value!

Focus on the UW Libraries Contribution to the Research Enterprise How does the library support sponsored research? What is the value of the library contribution? Some measures & indicators of library contributions Collections budget and usage data Survey information Bibliometrics Frequency of use Time saved/contingent valuation Information seeking behavior Qualitative information Services used Library research projects/collaborations

The Importance of the Research Enterprise University of Washington Operating Revenues $2.4 Billion in 2007-08 Research Grants $1.1 Billion Health and Human Services $510 million National Science Foundation $80 million Other federal agencies $200 million Industry/Foundations $110 million Other non-federal $160 million

University of Washington Libraries Assessment Methods Used Large scale user surveys every 3 years (“triennial survey”) beginning in 1992. In 2007: 1500 faculty responses; 600 grad; 500 undergrad In-library use surveys every 3 years beginning 1993 4000 surveys returned in 2008 Focus groups/Interviews (annually since 1998) Observation (guided and non-obtrusive) Usability Usage statistics/data mining Information about assessment program available at: http://www.lib.washington.edu/assessment/

The Changing Business Model: Trends in Library Use at the University of Washington UW Libraries Usage Data Items Used In-Library 800,000 in 2002-03 300,000 in 2007-08 (6.0 million article downloads in 2007-08) Gate Counts 4.6 million in 2002-03 4.3 million in 2007-08 (8.7 million Web site user sessions in 2007-08) In-Person Reference Questions 140,000 in 2002-03 90,000 in 2007-08 (18% of all ref queries are virtual in 2007-08) Traditional Library Core Business (Usually in-person) Physical Collections Print (primarily) Microforms Other Facilities House collections Customer service & work space Staff work space Services Reference Instruction Access

Qualitative Information Provides Context BioScience Interview/Focus Groups (2006) Content is primary link to the library Identify library with ejournals; want more titles & backfiles Provide library-related services and resources in our space not yours Discovery begins primarily outside of library space with Google and Pub Med; Web of Science also important Library services/tools seen as overly complex and fragmented Print is dead, really dead If not online want digital delivery Go to physical library only as last resort Personal connection is important

Off-Campus Remote Use 1998-2007 (Percentage using library services/collections at least 2x week) 76% of faculty (80% of those using federal research funds) connect online at least 2x week

Primary Reasons for Faculty Use of Libraries Web Sites by Academic Area (2007 Triennial Survey, at least 2x per week)

Importance of Books, Journals, Databases byAcademic Area Scale of 1 “not important” to 5 “very important)

UW Libraries Serial Purchasing and Usage 85% of collections budget spent on serials/backfiles Ranked 4th in 2006-07 ARL serial expenditures ($11.4 million) 6 million journal article downloads (Counter compliant) 30% increase over 3 years 75% of downloads from scholarly journals in science-engineering-health sciences Bibliographic database use has declined Most pronounced in subject specific databases Web of Science unchanged at 160,000 login sessions annually (13% of all log-in sessions)

UW Libraries Contribution to Sponsored Research Among faculty receiving federal research funding: 97% rate Libraries as very important to their work 96% rate journals as most important info resource for their work 93% rate the Libraries as making a major contribution to keeping current in their field 93% rate the Libraries as making a major contribution to their research productivity 80% connect to the Libraries at least twice per week from office, lab or off-campus

Differences in Library Contribution by Funding Source: (Scale of 1 “Low to 5 “High”)

Libraries Contribution to Your Being a More Productive Researcher by Academic Area and Funding Source

Closing the Loop: Success with Assessment Assess what is important Keep expectations reasonable and achievable Use multiple assessment methods; corroborate Mine/repurpose existing data Focus on users; how they work, find & use information Use the data to improve and add customer value Keep staff, customers and stakeholders involved and informed

Data trends can inform the future Eye to the Future Measuring performance is an exercise in measuring the past. It is the use of that data to plan an improved future that is all important. Peter Brophy Data trends can inform the future Strategic planning can frame the future Organizational performance models can align ongoing operations with future aspirations Understanding how customers work and how that work is changing is key to our future