1. Measuring Your Impact: Using Evaluation for Library Advocacy

2. Agenda MorningAfternoon 9:00 Introductions and Overview of Workshop 12:30LUNCH 9:20 Demonstrating the Library’s Value – Mission & Vision 1:30Data, Tools, and Analysis 9:50 The Environment (SWOT exercise) 2:45Communicating the Value 10:10 The Plan - Logic Model 3:15 Create your message – elevator speech, infographic, annual report introduction 11:00Break/Group exercise4:00 Resources, class questions and evaluation 11:30 The Plan – Evaluation Create your own, share 12:30LUNCH 2

3. Workshop objectives By the end of the workshop you will: Understand your library’s value in terms of the mission of the larger organization Describe some tools used to assess your library, its users and stakeholders Identify the tools and methods used for data collection and analysis Design an evaluation plan for a service in your library Communicate evaluation results 3

4. What this workshop is not… Custom-tailored A quick fix One size fits all 4

5. “What gets measured, gets done.“ Proving to your administration that money spent in your library is going to pay student achievement dividends is well worth the time and effort What gets measured gets done: http://www.doug-johnson.com/dougwri/what-gets-measured-gets- done.html 5

6. Demonstrating YOUR library’s value 6

7. Why demonstrate value? To show the impact of your library on the larger organization’s mission and goals To show accountability for your resources; your library’s contribution to the bottom line To demonstrate that your library provides value-added services To use as an advocacy and marketing tool 7

8. How & what to evaluate What you decide to evaluate depends on: – What you need or want to know – What your users feel is important – What certain stakeholders want to have evaluated Use “smallball” evaluation – Base hits or home runs? 8

9. Library service is value-added Cost savings – Institutional & Personal Time savings – Travel, discovery, skills acquisition, opportunity for other uses of time Access – Opportunity to use/apply information Impact on learning, research 9

10. The bottom line You are either generating revenue, or supporting those who do OR … You are helping to control operating expenses or supporting those who do OR … You are creating expenses that add recognized value OR … You are creating expenses that must be controlled or eliminated to reduce overhead Will Welton, PHD, Director, MHA Program, University of Washington 10

11. The big question! What is the bottom line and what is your library doing to support it? 11

12. What you need is a plan Know the vision Look at the environment Create the plan Collect and analyze the data Communicate the value 12

13. The vision 13

14. Know what you want to do and why Align your library’s mission & goals with the organization’s mission & goals Your library’s value = library’s contribution to achieving organization’s mission & goals 14

15. 15 Put library mission here Put institutional mission here Mission statements

16. Put another institutional mission here Put another library mission here 16 Mission statements (2)

17. haiku Mission haiku Mission 17 Haiku

18. Write a haiku Write your mission statement or another message Turn that into a haiku Standard haiku is 5/7/5 syllables, but English haikus are more flexible because of our syllable structure Share your haiku 18

19. Made to stick SUCCESs – Simple – core, compact – Unexpected – gets attention – Concrete – specific, not abstract – Credible – testable – Emotional – more than self interest – Stories – stimulate, inspire (From: Made to Stick. Heath, Chip. New York: Random House, 2007) 19

20. The environment 20

21. Why look at the environment? Understand needs, desires and problems in context Validate assumptions about your contributions and services Provide a baseline for future evaluation Help to develop the blueprint to plan and evaluate your contributions and services 21

22. Components of environment The organization Your library Clients/users Stakeholders The community 22

23. Assessing the environment User/stakeholder input – Surveys, focus groups, interviews – Unsolicited feedback – Conversations Observation – Assessment by walking around – Know the language of your users Information/literature Journal/serial/magazine 23

24. Assessing the environment (2) Library statistics and records – Collection use – Librarians’ work including reference, teaching, etc. Get out of your library and talk to people – Understand the role of information in their work – Learn how information is acquired, applied – Discover information problems they are trying to solve SWOT analysis – Understanding strengths, weaknesses, opportunities, threats – Consider how these affect your work, your assessment 24

25. Top 10 tasks - Librarians School Librarians say… 1.Help students select books 2.Collaborate with teachers 3.Read and book talk with classes 4.Teach research and use of technology to students and teachers 5.Keep library organized including cataloging, weeding, inventory, late notices, vacuuming, dusting 6.Study standards to prepare library lessons 7.Troubleshoot technology issues around the building 8.Help with extracurricular activities including laminating, school plays, lunch duty, etc 9.Prepare book, equipment and supply orders 10.Organize special library events like book fairs, author visits, contests 25

26. Top 10 tasks - Principals School Principals say… 1.Help students access information and books 2.Help faculty access information and books 3.Share technology expertise with students and teachers 4.Select “appropriate” materials 5.Model love for reading 6.Collaborate with teachers 7.Provide equipment and technology 8.Provide leadership with technology 9.Teach research skills, teach about books and databases 10.Provide an inviting environment How does your boss see you? Proof that principals value librarians. School Library Journal http://www.slj.com/2012/09/careers/how-does-your-boss-see-you-proof-that-principals-value-librarians/#_ 26

27. wordls Librarians say Principles say 27

28. SWOT analysis Strengths  Internal ◦ Positive statements about your library Weaknesses  Internal ◦ Statements about what is lacking in your library Opportunities  External ◦ What do stakeholders want you to do that you are not doing? Threats  External ◦ Factors that can adversely impact your library’s goals 28

29. Plan backward, implement forward 29

30. The logic model - a tool to… Help organize your thoughts – both before and during a project/program Provide a framework for planning and evaluating programs Clarify intended outcomes Acts as a communication tool Good source for more information: W.K. Kellogg Foundation Logic Model Development Guide (pdf on your thumb drive; see class bibliography for link) 30

31. Pieces of the logic model Outcomes Activities Resource Outputs 31

32. Outcomes Outcomes - changes in attitude, behavior, skills, knowledge or situation – Short, medium and/or long term – Intended or unintended – Positive as well as negative – S.M.A.R.T. 32

33. SMART outcomes Outcome: Health resources Web pages are available on the library website by the end of the project – Specific: one or more web pages with links to EBM resources are available on the library website Measurable: done or not; do nurses access resources Action-oriented: created and posted; nurses access resources Realistic: It is possible within the timeframe Timed: “when” is articulated 33

34. Outcomes examples Outcome – Long Term: Patients and their families will have improved health information literacy – Intermediate Term: Staff will have information for timely clinical decision-making – Short Term: Health resources Web pages are available on the library website by the end of the project 34

35. Activities and resources Activities – What will you do? Resources – What you have Income Equipment Collection – What you need Operating expenses (e.g., personnel, acquisitions, maintenance, etc.) Funds for new initiatives or services 35

36. Outputs – How many did you do? – How many attended? – How many were distributed? – How many times was it used? 36

37. Putting it all together Resources If we get these resources… Activities/Outputs Conduct these activities and deliver these products Outcomes We will see these outcomes 37

38. The model may change over time 38

39. Logic model form ResourcesActivityOutputsOutcomes Personnel, money, expertise needed What you will doWhat your activity will produce – data, classes, brochures, etc. The SO WHAT & WHEN – the benefits that accrue as a result of your program Add as many lines as needed to describe your plans 39

40. Logic model example ResourcesActivityOutputsST/IT/LT Personnel, money, expertise needed What you will doWhat your activity will produce – data, classes, brochures, etc. The SO WHAT & WHEN – the benefits that accrue as a result of your program Knowledgeable librarianSelect web resourcesList of URLs and descriptions of resources/sites HTLM expertiseCreate web page(s)Write HTML code or edit a template/existing page(s) Health resources Web pages are available on the library website by the end of the project Tracking softwareCollect page hit statistics; survey staff about usefulness, timeliness of web based information Web statistics; survey results Staff will have information for timely clinical decision-making Outcomes: ST: Health resources Web pages are available on the library website by the end of the project; IT: Staff will have information for timely clinical decision-making 40

41. Logic model group activity Identify outcome(s) Articulate short/intermediate/long term List – activities – outputs – resources 41

42. The evaluation plan 42

43. The plan Builds on the logic model Evaluates success Assesses value 43

44. Consider the purpose Who is your audience? This may be different than the beneficiaries of your services – Your users – Administrators – Patients/families How will the information be used? – Financial savings or justification – Intangible or non-monetary value of program benefits to community – Marketing and advocating for your library 44

45. What do you want to know? Have you achieved your stated outcome? – Short Term: Health resources Web pages are available on the library website by the end of the project – Intermediate Term: Staff will have information for timely clinical decision- making – Long Term: Patients and their families will have improved health information literacy 45

46. When do you want to know? Consider objective(s) Articulate the time frame 46

47. Evaluation plan form Outcome (from logic model): Staff will have information for timely clinical decision-making ActivityIndicatorTargetData SourceData Analysis Methods Data Collection Frequency What is done to achieve outcome – from Logic Model Measurable result of activity Specific indicator measurement desired Origin of indicator measurements Organize, examine, learn from the data Date, time and intervals Add as many lines as needed to describe your plans 47

48. Evaluation plan example part 1 Outcome (from logic model): Staff will have information for timely clinical decision-making ActivityIndicatorTarget What is done to achieve the outcome Definition of successSpecific desired result Create web pages; analyze use statistics; survey potential users Web pages are available; Users indicate the information was available for clinical decision making 50% of respondents report information was available for timely [define timely here] clinical decision- making 48

49. Evaluation plan example part 2 Data SourceData Collection Frequency Data Analysis Methods Origin of indicator measurements Date, time and intervalsOrganize, examine, learn from the data Web statistics, surveyData recorded quarterly throughout fiscal year; twice annual survey Spreadsheets; survey coding 49

50. Now design your own evaluation plan 50

51. From your logic model… Select an outcome and describe: – An activity that will help achieve the outcome – Indicators of outcome results – Targets for indicators – Data source – Data collection frequency – Data analysis methods 51

52. Making sense of the data 52

53. Analysis Think about data analysis before collecting the data Identify experts that can help with data analysis Test your collection and analysis tools Leave enough time Go “smallball” if needed 53

54. Your data Quantifiable – Statistics What do you collect? What do you do with it? – Evidence of change – Measurable impact of services provided Qualitative – Stories, opinions – Evidence of change – Perceived impact of services provided 54

55. Some tools Benchmarking – How you compare to others of similar size and circumstance – A common measuring stick to evaluate process performance Qualitative/quantitative analysis – Coding narratives – Scoring responses Cost/Benefit Analysis – Use to evaluate the benefits of a program or service Return on Investment – How much your investment in the program or project earned for the company 55

56. Benchmarking Can improve your library's performance Can help you gain upper management support Can help prove the value of your library 56

57. Benchmarking resources Your own library The competition – Other libraries in your system, industry – Associations: – ARL – LibQual+ – AAHSL – Annual survey – NN/LM MAR library value study 57

58. Qualitative and quantitative Narrative – Code text for concepts – Determine frequency of concepts – Describe/illustrate results Quantitative – Assign values to possible responses – Calculate mean, median, mode of response values 58

59. CBA: cost/benefit analysis The ratio showing dollar value of benefits gained for dollar value of costs Use to evaluate the benefits of a program or service Benefits divided by costs gives the value realized by transaction 59

60. CBA formula BENEFITS ÷ COSTS Or BENEFITS COSTS 60

61. ROI: return on investment Percentage showing the return (increase in value) on dollars spent to achieve a benefit – Amount the company’s investment in a program earned for the company – Earnings on money spent 61

62. ROI formula ((BENEFITS – COSTS) ÷ COSTS) × 100 62

63. Preparing for CBA and/or ROI Select the program or service to value – Journal subscriptions; searching the literature class Identify benefits derived from program/service – Users have access to literature for decision making Identify who gets the benefit – Library users, patients, care team Convert the benefits to quantifiable terms (based on surveys and/or records) – # of journals used; # of articles accessed; time available for research, patient care 63

64. Examples of Benefits Books used (in house or borrowed) Print journals used (in house or borrowed) E-journal articles accessed Patron time saved Value of information available for research, teaching, patient care 64

65. Costs Purchase price Staff time Space, light, heat, computers, benefits 65

66. Costing tips Consider tangible vs. intangible benefits Be conservative in estimating benefits and liberal in assessing costs Obtain administrative approval of the value of the benefits for each alternative; involve financial staff Express benefits and costs in a value system shared by all parties – $$ Think about inflation and depreciation if study extends over several years – only if the CFO insists! 66

67. A CBA/ROI Example Subscription to online Fictional Medical Journal at Anytown Institution 67

68. Literacy class The library spends $1,600 for an institutional subscription to Fictional Medical Journal (FMJ) online and $200 to manage the subscription 7,800 articles are viewed per year using the library’s subscription 68

69. Assumptions Each person reads on average 2 articles per week – 2x52=104 articles per user 7,800 uses represents 75 users – 7,800/104=75 Without a library, each person who needed FMJ would subscribe at a cost of $99 for online only 69

70. Costs and benefits Cost: $1,800 – $1,600 for subscription and $200 for staff time to process/maintain Benefit: $7,425 – 75 personal subscriptions: 75 x $99 70

71. CBA math BENEFITS COSTS $7,425 $1,800 = $4.125 $4.13 benefit to the institution for each dollar spent by the library on FMJ 71

72. ROI math Benefits – Net Costs Net Costs × 100 $7,425 – $1,800 $1,800 × 100 = 312.5% 72

73. Replacing your library or a library service What is the value of the resources and services your library provides? What would your users pay on the open market? Would your institution pay the retail costs? Would your institution pay for everyone who needs information or for select staff? Retail Value Calculator: http://nnlm.gov/mcr/evaluation/calculator.html 73

74. Calculators that do the math for you Retail value calculator http://nnlm.gov/mcr/evaluation/calculator.html CBA/ROI calculator – CBA: Benefits/Costs – ROI: ((Benefits-Costs)/Costs) *100 http://nnlm.gov/mcr/evaluation/roi.html 74

75. Value calculator 75

76. Calculator data points Where do the values come from? What does each line mean? http://nnlm.gov/mcr/evaluation/valuation.html 76

77. Becker Library 77

78. ROI calculator 78

79. ROI for Becker Library 79

80. CBA/ROI exercise Select one service or resource List all the costs associated with providing the service once or obtaining one resource List alternative sources for that service or resource List the costs associated with that alternative source Plug the numbers into the formula 80

81. Communicating the Results 81

82. Use the results… To communicate your value To promote and advocate for your library To maintain or get additional funding As baseline information to show improvement over time To improve library services and programs 82

83. Some communication methods Written and/or oral reports – Formal – Informal Graphics Executive summary Press release for newsletter or local paper Promotional materials for your library Staff meetings Informal hallway or elevator talk 83

84. Made to stick (redux) SUCCESs – Simple – core, compact – Unexpected – gets attention – Concrete – specific, not abstract – Credible – testable – Emotional – more than self interest – Stories – stimulate, inspire (From: Made to Stick. Heath, Chip. New York: Random House, 2007) 84

85. Becker Library annual report 85

86. 86 Graphic for illustrating vision

87. Bookmark example 87

88. Ad for library services 88

89. Librarian roles posters 89

90. Tips for writing Frame message in terms of added value to the mission and goals of the organization Summarize in one paragraph then support the message with evidence. Use figures, tables, graphs to make your point – Should be able to stand alone – Have a narrative to give more details Have others review the communications 90

91. Tips for talking Know and rehearse your message Talk to administrators one-on-one when possible Focus on benefits Don’t use library jargon Educate, but don’t bore 91

92. “Elevator talk” Simple, memorable mini-speech Ready for any brief encounter What your library contributes and why that matters Create curiosity “Sadly and too often, the people who have the most influence over our future are really not all that interested in libraries.” Gardner, Reed S. Making the Case for Your Library: A How-To-Do-It Manual. How-To-Do-It Manuals for Librarians, Number 104. New York: Neal-Schuman, 2001: 6 92

93. Elevator talk exercise Select an outcome Invent results Determine the audience Create – a sound bite that highlights the value of the outcome to that audience OR – an interesting but mysterious statement that will make your listener want to know more 93

94. Go forth & evaluate Do smallball evaluation Relate your value to the mission and goals of your organization Plan your evaluation before you begin Share your value with many audiences Don’t wait for a crisis – start now! 94