1. Assess for Success: Proving Library Value
Steve Hiller
Director, Planning and Assessment University of Washington Libraries

2. Assessment Basics
What do we need to know about our communities and customers to make them successful?
How do we measure the effectiveness of our services, programs and resources?
What do our stakeholders need to know in order to provide the resources needed for a successful library?

3. Assessment Is More than Numbers
Library assessment is a structured process:
To learn about our communities
To respond to the needs of our users
To improve our programs and services
To contribute to the success of our communities

4. Assessment Process
Focuses on understanding customer needs and offering services that meet those needs
Collects, analyzes and uses quantitative and qualitative data for more effective management and decision-making
Emphasizes ongoing communication with customers and stakeholders,
Seeks opportunities for collaboration and comparisons within the organization, institution and beyond

5. What Do We Need to Know About Our Customers?
Who are our customers (and potential customers)?
What areas/fields/courses are they working in?
How do they work? What’s important to them?
How do they find information needed for their work?
How do they use our services? What would they change?
How do they differ from each other in use/needs?
Outcomes
How do we add value to their work?
How do we contribute to their success?

6. The Value of Community Assessment
Identify actual and potential customers
Understand needs and use preferences
Use funding and staff effectively
Understand and address “competition”
Encourage community involvement and “ownership”
Target marketing, market penetration and outreach
Measure, demonstrate, present the value of the library to the community and stakeholders

7. Thinking Strategically About Library Futures
What is our central work and how can we do more, differently, and at less cost?
What important services do we provide that others can’t?
What advantages do we possess?
How is customer behavior changing?
How do we add value to our customers work?
What are the essential factors responsible for library success now and in the future?

8. The Challenge for Libraries
Traditional statistics and data are no longer sufficient
Emphasize inputs – how big and how many
Do not tell the library’s story
May not align with organizational goals and plans
Do not measure service quality
Need data from the user’s perspective
Need to provide organizational accountability to funders and other stakeholders
Need to define where we want to be in 3/5/more years
Need to answer: What value do we provide our primary community and how do we show it?

9. Defining Success in a Digital Environment
Crafting new indicators and measures of success
Moving from measuring inputs to outputs and outcomes
Understanding impact of library/partner roles and services on the community
Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing community goals.
Reallocating resources and managing capabilities to achieve success.
Demonstrating value of library contributions

10. Some New Metrics for Academic Libraries
Uniqueness of collections
Value of consortia
Administrative and budgetary efficiencies
Student outcomes/student learning/graduate success
Contributions to faculty productivity
Social frameworks/intellectual networks
Generating new knowledge
Creating the collective good with reusable assets such as an institutional repository or e-science

11. Good Assessment Starts Before You Begin . . . Some Questions to Ask
Define the question
What do you need to know and why
How will you use the information/results
Where/how will you get the information
What are appropriate methods
Is there existing information
New data (where or who will you get it from)
Is it cost and resource effective
How will you analyze the information
Who will act upon the findings

12. How Do We Get Customer Information?
Statistics/data mining (local, institutional)
Surveys
Focus groups
Observation
Usability
Interviews
Embedding
Logged activities
Comments, suggestions, “over the counter”, being there

13. Another View of Metrics

14. Getting The Right Data Isn’t Enough
“…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.”
Carl Becker
Annual Address of the President of the American Historical Association, 1931
Understanding what the data means and how to follow-up are crucial

15. Make Data Meaningful Summarize Compare Analyze Present
Go below the surface to examine results by:
Demographic group
Users and non-users
Stakeholders vs non-stakeholders
Compare current data with past; look for trends
How can we use the data for action?

16. Data Presentation: 3 Key Questions
What’s the message?
Fewer “messages” means greater impact
Who’s the audience?
Multiple audiences may need different presentations
How do we present?
Quantitative data - Be graphic , provide meaning/understanding
Qualitative data - Be selective, provide corroborating data
Keep it simple AND focused!

17. And Remember the Content!

18. Moving from Analysis to Action
From all of the data, determine what can and should be addressed
Prioritize the important/critical action items
Make evidence-based decisions
Align with mission, vision and goals of parent organization
Address what is important to customers
Establish action guidelines
Who, how, when

19. What Makes it Hard to Be Evidence-Based
What Makes it Hard to Be Evidence-Based? (From Pfeffer and Sutton, 2006)
There’s too much evidence
There’s not enough good evidence
The evidence doesn’t quite apply
People are trying to mislead you
You are trying to mislead you
The side effects outweigh the cure
Stories are more persuasive anyways

20. When the Evidence Isn’t Used

21. Collection Assessment: The 4 Questions
What’s important from the: Library perspective? Customer perspective? Stakeholder perspective? Vendor/publisher perspective(s)?

22. Some Good Reasons to Assess Collections
Meeting user needs
Costs/Budget/Space
Changes in the information discovery/use processes
Accountability to stakeholders, funding agencies
Breadth and depth; comprehensiveness
Environmental changes (social, information, technical)
Changes in publisher/vendor marketing and packaging
Return on Investment (ROI); contingent valuation

23. What Numerical/Cost Data Do We Need?
Costs
Collections (annual/over time)
Titles
Packages, consortia
Operational
Staff
Facilities
Vendors
Usage
Print (on-site, check-out, ILL)
Electronic/online
Type of use
Packages, consortial, doc delivery
Change over time
Cost per use
What type of use
Downloads
Searches
Loans
Page views
Packages, individual titles
Global information
Collection comparisons
Publishing output
Bibliometric tools, citation analyses, collection
Community demographics

24. So We Need Systems/Products That:
Enable us to track and validate usage
Enable us to track item level and package level costs
Provide comparative information
Provide “canned”/custom reports easily
Provide data in flexible formats we can manipulate/analyse
Provide information that meets established standards
Present snapshot and trend data
Are cost-effective

25. E-Metrics Tools: COUNTER, SUSHI and ERMI
Counting Online Usage of NeTworked Electronic Resources
Standardized way to count usage
Defined reports for journals, databases, and eBooks
Standardized Usage Statistics Harvesting Initiative
Model for automation of statistics harvesting
SUSHI feed: usage data that magically appears!
Electronic Resources Management Initiative
Can accept SUSHI feeds
Easy to generate COUNTER-style reports within ERM module
Brings in data from order records to calculate cost per use

27. Genesis Statistics Detail
Title
2007 CPU
2007 Use
2008 CPU
2008 Use
Genesis
$3.00
649
$3.67 93
2007 payment / 2007 Use = 2007 CPU
$ / 649 = $3.00

28. Don’t Forget the Qualitative
Discovery pathways and user expertise
Usability
User needs
User outcomes
What did the information enable the user to accomplish?

29. Integrated Collection Assessment
Collections-centered
Size, growth, formats, authors, subjects, comparisons
Cost-centered
Expenditures, price, inflation, cost per use, return on investment
Use-centered
Loans, e-usage, purchase requests, ILL
Customer centered (secondary)
Community demographics, programs, research, bibliometric studies, consortial
Customer centered (primary)
Behavior, impact, outcomes, success, discovery

30. What Will We Do With the Information?
Analyze and understand
Present and share
Make good decisions and choices
Ensure we add value to the work of our community
Customers and stakeholders make the ultimate determination of value!

31. Focus on the UW Libraries Contribution to the Research Enterprise
How does the library support sponsored research?
What is the value of the library contribution?
Some measures & indicators of library contributions
Collections budget and usage data
Survey information
Bibliometrics
Frequency of use
Time saved/contingent valuation
Information seeking behavior
Qualitative information
Services used
Library research projects/collaborations

32. The Importance of the Research Enterprise University of Washington Operating Revenues $2.4 Billion in
Research Grants $1.1 Billion
Health and Human Services
$510 million
National Science Foundation
$80 million
Other federal agencies
$200 million
Industry/Foundations
$110 million
Other non-federal
$160 million

33. University of Washington Libraries Assessment Methods Used
Large scale user surveys every 3 years (“triennial survey”) beginning in In 2007:
1500 faculty responses; 600 grad; 500 undergrad
In-library use surveys every 3 years beginning 1993
4000 surveys returned in 2008
Focus groups/Interviews (annually since 1998)
Observation (guided and non-obtrusive)
Usability
Usage statistics/data mining
Information about assessment program available at:

34. The Changing Business Model: Trends in Library Use at the University of Washington
UW Libraries Usage Data
Items Used In-Library
800,000 in
300,000 in
(6.0 million article downloads in )
Gate Counts
4.6 million in
4.3 million in
(8.7 million Web site user sessions in )
In-Person Reference Questions
140,000 in
90,000 in
(18% of all ref queries are virtual in )
Traditional Library Core Business (Usually in-person)
Physical Collections
Print (primarily)
Microforms
Other
Facilities
House collections
Customer service & work space
Staff work space
Services
Reference
Instruction
Access

35. Qualitative Information Provides Context BioScience Interview/Focus Groups (2006)
Content is primary link to the library
Identify library with ejournals; want more titles & backfiles
Provide library-related services and resources in our space not yours
Discovery begins primarily outside of library space with Google and Pub Med; Web of Science also important
Library services/tools seen as overly complex and fragmented
Print is dead, really dead
If not online want digital delivery
Go to physical library only as last resort
Personal connection is important

37. Off-Campus Remote Use 1998-2007 (Percentage using library services/collections at least 2x week)
76% of faculty (80% of those using federal research funds) connect online at least 2x week

38. Primary Reasons for Faculty Use of Libraries Web Sites by Academic Area (2007 Triennial Survey, at least 2x per week)

39. Importance of Books, Journals, Databases byAcademic Area Scale of 1 “not important” to 5 “very important)

40. UW Libraries Serial Purchasing and Usage
85% of collections budget spent on serials/backfiles
Ranked 4th in ARL serial expenditures ($11.4 million)
6 million journal article downloads (Counter compliant)
30% increase over 3 years
75% of downloads from scholarly journals in science-engineering-health sciences
Bibliographic database use has declined
Most pronounced in subject specific databases
Web of Science unchanged at 160,000 login sessions annually (13% of all log-in sessions)

41. UW Libraries Contribution to Sponsored Research
Among faculty receiving federal research funding:
97% rate Libraries as very important to their work
96% rate journals as most important info resource for their work
93% rate the Libraries as making a major contribution to keeping current in their field
93% rate the Libraries as making a major contribution to their research productivity
80% connect to the Libraries at least twice per week from office, lab or off-campus

42. Differences in Library Contribution by Funding Source: (Scale of 1 “Low to 5 “High”)

43. Libraries Contribution to Your Being a More Productive Researcher by Academic Area and Funding Source

44. Closing the Loop: Success with Assessment
Assess what is important
Keep expectations reasonable and achievable
Use multiple assessment methods; corroborate
Mine/repurpose existing data
Focus on users; how they work, find & use information
Use the data to improve and add customer value
Keep staff, customers and stakeholders involved and informed

45. Data trends can inform the future
Eye to the Future
Measuring performance is an exercise in measuring the past. It is the use of that data to plan an improved future that is all important. Peter Brophy
Data trends can inform the future
Strategic planning can frame the future
Organizational performance models can align ongoing operations with future aspirations
Understanding how customers work and how that work is changing is key to our future