1. Collaborative Assessment: Using Balanced Scorecard to Measure Performance and Show Value Liz Mengel, Johns Hopkins University Vivian Lewis, McMaster University Northumbria Conference (August 25, 2011)

2. Session Overview 1.The ARL Balanced Scorecard Project 2. Measures Commonality 3.Synergies with ARL Statistics Program 4. Ongoing Research Project

3. Copyright © 2009 Ascendant Strategy Management Group Balanced Scorecard Mission What is our plan to achieve our mission and vision? Mission What is our plan to achieve our mission and vision? Financial Perspective "If we succeed, how will we look to our donors or taxpayers?” Financial Perspective "If we succeed, how will we look to our donors or taxpayers?” Customer Perspective "To achieve our mission, how must we look to our customers?” Customer Perspective "To achieve our mission, how must we look to our customers?” Internal Perspective "To satisfy our customers and financial donors, which business processes must we excel at?” Internal Perspective "To satisfy our customers and financial donors, which business processes must we excel at?” Learning and Growth Perspective "To achieve our mission, how must our organization learn and improve?” Learning and Growth Perspective "To achieve our mission, how must our organization learn and improve?” Source: Strategy Maps© 2004, Robert S. Kaplan / David P. Norton

4. Balanced Scorecard Mission Objectives (Strategy Map) MeasuresInitiatives

5. ARL Balanced Scorecard Project Explore suitability of BSC for academic research libraries Encourage cross-library collaboration Explore concept of common objectives and metrics emerge?

6. ARL Balanced Scorecard Timeline 2009 – ARL call for participants 2010 – Training, initial implementation 2011 – Implementation, Refinement, Research Cohort #1 – ARL call for Cohort #2

7. Cohort 1 Common Objectives Financial Perspective Objective – Secure funding for operational needs (4/4) Customer Perspective Objective – Provide productive and user centered spaces, both virtual and physical (4/4) Learning and Growth Perspective Objective – Develop workforces that are productive, motivated, and engaged (4/4) Internal Processes Perspective Objective – Promote library resources, services, and value (3/4)

8. Copyright © 2009 Ascendant Strategy Management Group Measures/Perspective Mission Financial Perspective 19 Measures Financial Perspective 19 Measures Customer Perspective 41 Measures Customer Perspective 41 Measures Internal Perspective 19 Measures Internal Perspective 19 Measures Learning and Growth Perspective 15 Measures Learning and Growth Perspective 15 Measures Source: Strategy Maps© 2004, Robert S. Kaplan / David P. Norton Numbers indicate the combined measures of 3 of the Cohort 1 libraries

9. A measure by any other name…

10. Learning about measures

11. Year 1 Measures Utilizing ARL Standard Statistics JHU, McMaster, UW9.5% (9/94) ARL Statistics or Supplemental Statistics8 ARL Investment Index1

12. Year 1 Measures Utilizing Surveys JHU, McMaster, UW23% (22/94) LibQUAL4 ClimateQUAL2 WOREP (Wisconsin Ohio Reference Evaluation Program) READ Scale (Reference Effort Assessment Data) 1 Locally Created15

13. Qualitative Coding Year 1 Measures Collections 23 Financial/Funding 13 Instruction 13 Library Staff 11 Library Services 8 Space 5 User Assessment 5 Balanced Scorecard Effectiveness 3 Donors/Development 3 Exhibits/Museums 3 Liaison Services 2 Web 2 Data Curation 1 IT problems 1 Promotion/Marketing 1

14. Fluidity Many iterations before measures are “finalized” McMaster example The refinement process is exhausting!

15. The Impact of Collaboration Reviewed each others’ slates Borrowed formulas, approaches to gathering data, etc. Benefits Cohort 1: – time – credibility – benchmarking

16. Research Objective Explore two options for furthering the concept of collaborative scorecard development an inventory of all used measures a common set of core measures

17. Research Questions Perceived Usefulness 1.Will ARL scorecard planning teams perceive the provision of a standardized set of core measures or an inventory of used measures as helpful to their local implementations? 2.Of the two options, which is perceived as most helpful? Measure Adoption 3.Were measures added, deleted or changed as a result of the set sharing activity?

18. Why is this important? To facilitate inter-university comparisons To facilitate scorecard implementation at local sites (save time, share expertise…) To define common and distinctive capacities.

19. OPTION ONE : The Inventory A compilation of ALL measures in use across the ARL sites – arranged by common theme. EXAMPLE: Instruction Effectiveness – “% of classes incorporating feedback and receiving a 4 or 5 from faculty at the end of the semester.” – “% change on pre-test/post-test for intersession, EWP, and museums' instruction sessions.” – “class evaluations will indicate consistent or improved overall ratings for library instruction sessions.”

20. OPTION TWO: A “Common” Set A set of common measures (based on Cohort 1 slates) – using consistent measure names and formulas. EXAMPLE: Job Satisfaction Measure – “Increase in the percentage of staff answering ** or above in ClimateQUAL question #***.”

21. Research Plan Time LineEvent May - 2011Cohort #2 begins training & scorecard development June – November 2011Proposal Refinement -Discuss with Cohort #1 (July) -Present at Northumbria (August) -Present at ARL Statistics & Assessment Committee (Oct.) Nov. 2011 – Dec. 2011Creation of the 2 sets (the “common set” and the “inventory”) Jan. (late) 2012Presentation of 2 tools to Cohort #2 (ALA MidWinter) March 2012Telephone Interviews with Cohort #2 Planning Teams June 2012Telephone Focus Group Discussion with Cohort #2 Review measures proposed by teams. September 2012Write up results October 2012Present preliminary results at Library Assessment Conference Charlottesville, Virginia

22. Vivian Lewis lewisvm@mcmaster.ca Liz Mengel emengel@jhu.edulewisvm@mcmaster.caemengel@jhu.edu