1. www.arl.org Making Library Assessment Work ARL 4th Human Resources Management Symposium Washington, D.C. November 9, 2004 Steve Hiller and Jim Self University of Washington and University of Virginia Association of Research Libraries

2. www.arl.org Why Assess? Accountability and justification Improvement of services Comparison with others Identification of changing patterns Identification of questionable services Marketing and promotion

3. www.arl.org Good assessment practices Focus on the user Diverse samples of users Fair and unbiased queries Measurable results Criteria for success Qualitative and quantitative techniques Corroboration

4. www.arl.org Assessment is not… Quick and easy Free and easy A one-time effort A complete diagnosis A roadmap to the future

5. www.arl.org “…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.” Carl Becker Annual Address of the President of the American Historical Association, 1931

6. www.arl.org What Does it Mean? Understanding Your Data Scan results for basic overview – Frequencies, means, patterns, variation Use statistical analysis that make sense Qualitative information and comparisons provide context and understanding Seek internal or external validation – Within same data sets or others Identify what is important and why

7. www.arl.org Communicating and Using Results Identify key findings, not all results Mix text, data, and graphics – avoid jargon – add context Know your audiences. Make it understandable Prioritize potential action items and follow-up Identify “Handoffs” to those responsible for action Look for some easy “wins” – Quick, inexpensive, and noticeable Report results

8. www.arl.org Effective Assessment Easier Said Than Done Libraries in many cases are collecting data without really having the will, organizational capacity, or interest to interpret and use the data effectively in library planning. The profession could benefit from case studies of those libraries that have conducted research efficiently and applied the results effectively. (Denise Troll Covey, Usage and Usability Assessment: Practices and Concerns, 2002)

9. www.arl.org Two Approaches to Assessment University of Washington – User needs assessment – Large-scale cyclical surveys and ongoing qualitative input – Assessment distributed throughout organization University of Virginia – Performance and financial standards – Compilation of data from varied sources – Centralized Management Information Services unit

10. www.arl.org UW Libraries Assessment Organization Library Assessment Coordinator (50%) – Chairs Library Assessment Group (9 members) – Coordinates and develops broad-based user needs assessment efforts (surveys, focus groups, observation) – Encourages and supports other assessment work Shared and Distributed Assessment Activities Usability (Web Services) E-Metrics (Assessment, Collection Management Services) Management information (Assessment, Budget, CMS) Instruction (Information Literacy, Assessment) Digital Library (Digital initiatives, Public Services, Assessment)

11. www.arl.org UW Assessment Methods Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004 In-library use surveys every 3 years beginning 1993 LibQUAL+™ in 2000, 2001, 2002, 2003 Focus groups on varied topics (annually since 1998) Observation (guided and non-obtrusive) Usability E-Metrics

12. www.arl.org Growing Assessment at UW From Project-Based to Ongoing and Sustainable Libraries’ first strategic plan in 1991 called for survey as part of user-centered services philosophy – Initial large scale library survey done in 1992 as “one-time project” Library Services Committee formed in 1993 – Conducted in-library use surveys in 1993,1996, triennial survey in 1995 Library Assessment Group appointed in 1997 – Focus groups, observation studies, in-library and triennial surveys Collection Management Services, 1997 – E-Metrics and collections use Library systems, 1997 – Usability, Web usage logs Library Assessment Coordinator (50%) appointed 1999

13. www.arl.org UW Assessment Priorities Information seeking behavior and use Library use patterns Library importance and impact User priorities for the library User satisfaction with services, collections, overall Using data to make informed decisions that lead to library improvement

14. www.arl.org How UW Has Used Assessment Information Understand that different academic groups have different needs Make our physical libraries “student” places Identify student information technology needs Move to desktop delivery of resources Enhance resource discovery tools Provide standardized service training for all staff Stop activities that do not add value to users Consolidate and merge branch libraries

15. www.arl.org Branch Library Consolidation A UW Case Study 3 Social Science libraries consolidated in 1994 – Described in ARL SPEC Kit Review Committee formed in 2002 Changing use patterns a bigger driver than budget Review to be objective and data-based Identify one library to be consolidated into main library

16. www.arl.org Performance Measures to Assess Branch Library Viability Use – Print items, photocopies, reference questions, gate counts Primary user population – Number of faculty and students, change over time Facility quality – For users, collections, and staff Physical library dependency of primary users

17. www.arl.org Data Sources Used Library generated use data (including trend data) Electronic resources use data supplied by vendors University enrolment data (including trend data) Interviews, focus groups, survey comments Facility data Survey data – Triennial survey – In-library use Cost data

18. www.arl.org Primary User Groups by Branch Library (2003 University data)

19. www.arl.org Facility Space Quality: Methodology Discussed facility issues with unit staff Reviewed user survey comments from 2001 and 2002 Used previous focus group data for fine arts libraries Developed list of criteria A team of 3 walked through each unit A second walk through was conducted 2 months later Each member of the team assigned a score of 1 to 5 for quality of staff, collections, and user spaces. Scores were compared and made consistent.

20. www.arl.org Facility Quality

21. www.arl.org Science Faculty Libraries Used Regularly (2001 Survey) College/Dept’s (Faculty Responses) “HOME” Library Natural Sciences/Allen Other Chemistry (36) 72%33%28% (Physics) Fish-Ocean (42) 69%45%10% (Engineering) Forest Resources (28) 79%57%14% (Engineering) Math-Stat (27) 93%30%7% (Engineering) Physics-Astronomy (36) 78%25%22% (Chemistry)

22. www.arl.org Forestry Faculty and Grad Students Frequency of Library Use ModeGroup 2001 1x per week 2004 1x per week 2001 2 x+ per week 2004 2x+ per week Visit in Person FAC GRAD 29% 43% 24% 18% 0% 23% Campus Computer FAC GRAD 46% 53% 30% 18% 32% 35% 52% 71% Off-Campus Computer FAC GRAD 36% 24% 35% 53% 0% 12% 13% 24%

23. www.arl.org

24. Merger Time Line Review Group formed Spring 2002 Recommendations submitted February 2003 – Merger of Forest Resources Library – Identification of two other libraries for later merger Recommendation accepted June 2003 Joint implementation team appointed September 2003 Forestry faculty and students surveyed and presentation made February 2004 Forest Resources Library merged into Natural Sciences Library August 2004

25. www.arl.org Triennial Survey Spring 2004 Satisfaction All Faculty and Forestry Faculty (1998, 2001, 2004)

26. www.arl.org The University of Virginia Library Organizational Culture Customer Service Collecting and using data Innovation Flexibility Learning and development Participation and discussion Pride

27. www.arl.org In the words of our leader… Use data to IMPROVE Services Collections Processes Performance Etc., etc. Don’t use data to preserve the status quo -Karin Wittenborg University Librarian, University of Virginia June 24, 2004

28. www.arl.org University of Virginia Library Organizing for Assessment Management Information Services unit – Established in 1996 – Currently 3 staff – Resource for library management and staff – Advocates for sustainable assessment Centralized data collection, analysis and compilation Multifaceted approaches

29. www.arl.org Collecting the Data at U.Va. Customer Surveys Staff Surveys Mining Existing Records Comparisons with peers Qualitative techniques

30. www.arl.org Customer Surveys Faculty – 1993, 1996, 2000, 2004 Students – 1994, 1998, 2001, 2005 – Separate analyses for grads and undergrads

31. www.arl.org

32. Faculty Priorities 1993 to 2004

33. www.arl.org Using Customer Survey Results – UVa Additional resources for the science libraries (1994+) Major renovation (2001) Revision of library instruction for first year students (1995) Redefinition of collection development (1996) Initiative to improve shelving (1999) Undergraduate library open 24 hours (2000) Additional resources for the Fine Arts Library (2000) Support transition from print to electronic journals (2004)

34. www.arl.org Staff Surveys Internal Customer Service – 2002, 2003, 2004 – 1 to 5 satisfaction scale Worklife Survey – 2004 – Agree or disagree with positive statements

35. www.arl.org Internal Customer Service Surveys Ratings (1 to 5) of units providing service to other library staff Reports to managers and administrators Anonymous structured interviews to follow up Survey expanded in 2004 to include all library departments

36. www.arl.org Worklife Survey Areas of inquiry – Job Satisfaction – Interpersonal Relations – Communications & Collaborations – Diversity – Resource Availability – Staff Development – Health & Safety Report at Library ‘Town Meeting’ Focus groups following up

37. www.arl.org Data Mining Acquisitions Circulation Finance University Records

38. www.arl.org Acquisitions Expenditures by Format University of Virginia Library

39. www.arl.org University of Virginia Library Serving the Customer

40. www.arl.org University of Virginia Library Serving the Customer

41. www.arl.org Comparisons with Peers Within the University Within ARL

42. www.arl.org Expenditures of UVA Academic Division 1989 — 2003 Other Academic Support (+200%) Research (+219%) Total Academic Division (+140%) Libraries (+81%) Instruction (+80%)

43. www.arl.org Median Faculty Salaries University of Virginia Library Compared to ARL Median

44. www.arl.org Qualitative Techniques Focus Groups – Preparation for work life survey – Follow up to work life survey Structured Interviews – Anonymous follow-up to customer service survey Open Discussions

45. www.arl.org Corroboration Data are more credible if they are supported by other information John Le Carre’s two proofs

46. www.arl.org Analyzing Survey Results Two Scores for Resources, Services, Facilities – Satisfaction = Mean Rating (1 to 5) – Visibility = Percentage Answering the Question Permits comparison over time and among groups Identifies areas that need more attention

47. www.arl.org UVa Reference Activity and Reference Visibility in Student Surveys

48. www.arl.org The Balanced Scorecard Managing and assessing data The Balanced Scorecard is a layered and categorized instrument that – Identifies the important statistics – Ensures a proper balance – Organizes multiple statistics into an intelligible framework

49. www.arl.org The scorecard measures are “balanced” into four areas The user perspective The finance perspective The internal process perspective The future (learning and growth) perspective

50. www.arl.org Metrics Specific targets indicating full success, partial success, and failure At the end of the year we know if we have met our target for each metric The metric may be a complex measure encompassing several elements

51. www.arl.org Rationale for the BSC: Getting Control of the Data Focus Balance Assessment Intelligibility

52. www.arl.org The BSC at the U.Va. Library Implemented in 2001 Reports for FY2002 and FY2003 Reporting results for FY2004 Metrics for FY2005 in place A work in progress

53. www.arl.org Metric L.1.A. Impact of training. Target1: : Positive scores (4 or 5) on training statements from 80% of respondents in the annual work-life survey. Target2: : Positive scores (4 or 5) on training statements from 60% of respondents in the annual work-life survey.

54. www.arl.org Metric I.2.A. Internal Communications Target1: : Positive scores (4 or 5) on internal communications statements from 80% of respondents in the annual work-life survey. Target2: : Positive scores (4 or 5) on internal communications statements from 60% of respondents in the annual work-life survey.

55. www.arl.org Metric I.3.B. Staff Rating of Internal Services Target1: : A composite rating of 4.00 in the annual internal services survey, with no unit rated below 3.50. Target2: : A composite rating of 3.50, with no unit rated below 3.00.

56. www.arl.org Metric L.2.A. Job Satisfaction. Target1: : Positive scores (4 or 5) on job satisfaction statements from 80% of respondents in the annual work-life survey. Target2: : Positive scores (4 or 5) on job satisfaction statements from 60% of respondents in the annual work-life survey.

57. www.arl.org Metric L.2.B. Retention Rate of Employees Target1: : Retain 95% of employees. Target2: : Retain 90% of employees.

58. www.arl.org Metric L.2.C. Compare Salaries to Peer Groups Target1: : The median faculty salary at the U.Va. Library should rank in the upper 40% of all ARL university libraries. Target2: : The median faculty salary at the U.Va. Library should rank in the upper 50% of all ARL university libraries.

59. www.arl.org Metric L.2.D. Diversity of Staff Target1: : A net increase of at least 4 in faculty/staff diversity, with a net increase of at least 2 in faculty diversity. Target2: : A net increase of at least 2 in faculty/staff diversity, with a net increase of at least 1 in faculty diversity.

60. www.arl.org Metric L.4.A. Develop a Culture of Assessment Target1: : 75% of respondents score 12 or more positive responses on the Culture of Assessment IQ instrument. Target2: : 50% of respondents score 12 or more positive responses.

61. www.arl.org Moving Forward Understand your limitations – Use data wisely and appropriately – Don’t do more than you can support or utilize Don’t expect perfection; strive for accuracy and honesty Assess what is important to the library and the staff, Use the data to improve But always keep your focus on the user

62. www.arl.org For more information… Steve Hiller hiller@u.washington.edu www.lib.washington.edu/assessment/ Jim Self –self@virginia.eduself@virginia.edu –www.lib.virginia.edu/miswww.lib.virginia.edu/mis –www.lib.virginia.edu/bscwww.lib.virginia.edu/bsc ARL Assessment Project old.libqual.org/documents/admin/VPOHillerSelf.pdf