1. Quants, Quals, and Pals Library Assessment in the 21 st Century Steve Hiller University of Washington Libraries CUNY Assessment Conference June 6, 2014

2. Four Useful Assessment Assumptions Your problem/issue is not as unique as you think You have more data/information than you think You need less data/information than you think There are useful methods that are much simpler than you think Adapted from Douglas Hubbard, “How to Measure Anything” (2010)

3. What Will We Assess and Measure Institutional assessment efforts should not be concerned about valuing what can be measured, but instead about measuring what is valued. A. Astin Not everything that can be counted counts, and not everything that counts can be counted. Attributed to Albert Einstein

4. Some Questions Before You Start What Do You Need to Know and Why? What do you need to know Why do you need to know it How will you collect data (is it already available?) When do you need the data How will you analyze the data How will you use data What outcomes are expected The answers will determine methods used

5. Assess What’s Important Contribution to student and faculty success Contribution to institutional mission/visibility And, perhaps (depending on library and institution type) Accountability/Efficiency/Effectiveness Use Revenue generation (including fund raising) Benchmarking and comparisons with others Not how good is this library? Rather, 'How much good does it do?' (R.H Orr, 1973)

6. Recent Trends in Library Assessment and Performance Measurement Greater reliance on external measures and user impacts; aligned with planning Customer-centered library concept Outcomes-based assessment and metrics that made use of multiple methods, including qualitative Collaboration with institutional and other partners Demonstrating library impact and value on individuals and communities

7. Changing the Focus: Quants, Quals, Pals Quantitative – Focus changes from counting to analysis and action Qualitative – Focus changes from research study to practical methods which emphasize understanding Pals (Partners) – Focus changes from using our own data and internal assessments to working with other on broader collaborative assessments

8. Quants First mentioned in science fiction for those with exceptional quantitative skills and analysis SF Quants were often androids, or enhanced with implants or through breeding SF Quants, especially androids, had a high termination rate Quants became human (?) in the late 20 th century as a name for those with high analytical skills, especially in business

9. Quants in the 20 th Century: We Measure What We Can Count Inputs - Size – Collections – Budget – Staff – Facilities – Discovery tools Size used as indicator of excellence Outputs - Usage – Collections – Reference services – Instruction sessions – Facilities usage – Web sessions May indicate if inputs used but don’t tell us what users were able to accomplish

10. Data Accuracy

11. Many Library Statistics Are Self-Reported and Lack Independent Verification

12. The Challenge for Libraries Traditional statistics/measures are no longer sufficient – Emphasize inputs/outputs – how big and how many – Do not tell the library’s or customers’ story – Usually not aligned with organizational goals and plans or support library strategic directions and objectives – Do not capture the impact and value of the library Need to demonstrate the difference the library makes – To the individual, community and the organization

13. Why your kids will want to be data scientists Big money, lots of jobs, challenging and they’ll take care of you in old age!

14. Quants in the 21 st Century: Data Analysis and Action Finding meaning and application in: Big data sets Data mining Data exchange Actionable data Data formats Metadata Supported by sophisticated and robust data analysis tools

15. Some Quant Examples Infosphere use and application Relationship between library use and student, faculty and research success Data trends Data comparability and integration Analytics

16. Quals - Definition An abbreviation for the word Quality. Quals is used when the item being described surpasses the quality of everything else, by far, leaving only one word left - Quals. Only to be used by the super-cool and often used more frequently when drunk and rowdy.Qualitysuper-cool Source: Urban Dictionary

17. Classic Qual (itative) s Social Science research methodology using time- consuming protocols Interviews and observations (ethnographic) Focus groups Relatively few library qualitative studies until 25 years ago

18. Quals: The Past 25 Years Widely employed in libraries and for outcomes-based efforts Shorter, more focused and actionable studies Usability User-centered design Ethnographic Advocacy (narratives) Qualitative work integral part of market research

19. FutureQUAL® Merging quantitative and qualitative as data collection and analysis tools incorporate digital technology Understand individual and group behavior beyond information, libraries and organizations Narratives and personas closely linked to customized services, marketing and advocacy

20. Pals (Partners)

21. Partners and Collaborators Partners – Close, recognized relationships – Work towards similar goals and outcomes – Value each others contributions – Watch each others backs – Ride off in the sunset together Collaborators – Brought together for specific efforts – May have different goals – Bring special expertise – Watch your back! – Often shot at the end of spy/war movies

22. Partners and Collaborative Assessment: Collaboration with other institutional programs/units Collaboration with other libraries/consortia/ organizations Linked to strategic planning, accountability, advocacy Fostering assessment & performance measurement that focuses on community success

23. What Libraries Bring to Academic Partnerships Big picture understanding of higher education, scholarly communication, the institution and its programs Close links to academic programs through liaisons Service to entire community Student-centered approach and research collaboration Diverse skills and expertise

24. From Library-Centered to External Partnerships Research Traditional: Collection Development Collections most valued library research support. Remains important Now: Information Management Embed library experts within research programs Support research cycle from discovery through communication and archiving Collaborate with a variety of partners Instruction Traditional: Bib Instruction Library and tool focused Not always tied to course work or outcomes Evaluation by pre- and post-tests Now: Teaching & Learning Focus on student learning outcomes and authentic assessment Partnerships & contribution to teaching/program outcomes Broad use of educational technology

25. Catalyst for Partnerships Was Often External Association and information standards Integrated approach to accreditation (professional and institutional) Consortia Value and impact studies, advocacy

26. Library Value & Impact Lib-Value Project (ARL, Tennessee) Library Impact Data Project (Huddersfield/JISC) Methods and procedures for assessing the impact of libraries (ISO 16439) Discovering the Impact of Library Use and Student Performance (Wollongong) ACRL Assessment-in-Action project University of Minnesota Library Data & Student Success Value of Academic Libraries (Oakleaf, 2010)

27. Value & Impact: Key Themes Aligning library’s role with core mission & values of institution: “allow institutional missions to guide library assessment” (Oakleaf 2010, p, 30). Communicating value to stakeholders Connecting institutional & library data

28. U.S. Regional Accreditation Standards for Institutions of Higher Education NWCCU 2010 http://www.nwccu.org/index.htm http://www.nwccu.org/index.htm Institutional mission, core themes and expectations Resources and capacity – Governance – Educational resources – Library and information resources – Financial resources – Leadership and management – Infrastructure Planning and implementation Assessment Mission fulfillment, adaptation and sustainability

29. U.S. Regional Higher Education Accreditation: Library Standards (NWCCU) The institution holds or provides access to library and information resources with an appropriate level of currency, depth, and breadth to support the institution’s mission, core themes, programs, and services Planning for library and information resources is guided by data that include feedback from affected users and appropriate library and information resources faculty, staff, and administrators. The institution provides appropriate instruction and support to enhance efficiency and effectiveness in obtaining, evaluating, and using library and information resources that support its programs The institution regularly and systematically evaluates the quality, adequacy, and utilization of library and information resources and services, wherever offered and however delivered.

30. ACRL Standards for Libraries in Higher Education http://www.ala.org/acrl/standards/standardslibraries http://www.ala.org/acrl/standards/standardslibraries The Standards assume that libraries: Develop user-centered, measurable outcomes that articulate specifically what the user is able to do as an outcome Conduct assessments that may be quantitative and/or qualitative; Collect evidence from assessments that demonstrate degree of success Use assessment data for continuous improvement of library operations.

31. ACRL Standards for Libraries in Higher Education (2010) Institutional Effectiveness The library defines, develops and measures outcomes in the context of institutional mission and aligned with institutional outcomes The library develops and maintains a body of evidence that demonstrates its impact The library articulates how it contributes to student learning,

32. ACRL Standards: Libraries Contribution to Institutional Effectiveness The library contributes to student recruitment, retention, time to degree, and academic success. Students improve their academic performance over their college experience through their contact with the library. Students who have more contact with the library show higher levels of improvement in performance on standardized tests. Students are able to describe the role of the library in improving their academic performance from freshman to senior.

33. Partnerships in the 21 st Century Institutional teams working on institutional level issues with functional specialists Multi-institutional level networking and expertise sharing Defining institutional and programmatic goals, priorities, and outcomes Blurring of administrative and academic boundaries Computational savvy staff

34. Lorcan Dempsey’s 3 Challenges (2013) Engagement – Libraries work to create distinctive value in the research, learning and teaching workflows of their users in ways which go beyond the provision of collections Rightscaling – Libraries moving from “institution scale” to collaborative, shared systems – especially through functional and regional consortia. The need for local infrastructure declines. As libraries want to emphasize impact and engagement, and de-emphasize activities which do not create distinctive local value, rightscaling becomes a key question. Institutional innovation – Library as an organization which reconfigures to map changes in the user environment and expectations. Libraries have to develop new and routine ways of collaborating to achieve their goals, which involves evolution of organizational, cultural and communication approaches. – The learning that flows from institutional innovation

35. Four Library Assessment Questions What do we need to know about our communities and customers to make them successful? Who are our partners in collaborative assessment? How do we measure the effectiveness of our services, programs and resources and how they contribute to institutional and user success. What do our stakeholders need to know in order to provide the resources needed for a successful library?

36. 2014 Library Assessment Conference August 4-6, University of Washington http://libraryassessment.org/