1 Outcomes: Outcomes: Libraries Change Lives — Libraries Change Lives — Oh yeah? Prove it. Oh yeah? Prove it. The Institute of Museum and Library Services Public Libraries Association Phoenix, AZ 2002

2 Institute of Museum and Library Services The Institute of Museum and Library Services (IMLS) is a federal agency that fosters leadership, innovation, and a lifetime of learning through grants to museums and libraries. Please see IMLS’s Web site at for additional information about IMLS and its grant programs.

3 We will  Talk about outcome-based evaluation (OBE)  Talk about benefits  Define terms  Describe process  Discuss how to use results

4 Outcomes are achievements or changes in  Skill Information literacy Basket weaving Basket weaving  Knowledge A state’s population Symptoms of diabetes  Behavior Completes homework Reads to his kids What are outcomes?

5 Outcomes are achievements or changes in  Attitude Libraries are good I support recycling  Status High school graduate Certified librarian  Life condition Homeless Healthy What are outcomes?

6 Example Wally goes to a reading program and Learns childhood reading is important Learns childhood reading is important Wants to read to his son Wants to read to his son Uses a literacy program Uses a literacy program Advances 2 literacy levels Advances 2 literacy levels Gets his GED Gets his GED (What kinds of outcomes are each of these?)

7 Outcomes Where do they come from? Government Performance and Results Act (GPRA) 1993 Government Performance and Results Act (GPRA) 1993 New accountability, need to evaluate New accountability, need to evaluate Funding trends Funding trends – IMLS – State government, LSTA – Foundations (Gates, Pew)

8 Evaluate? Let me count the Ways! Internal Internal External External Participant Participant Process Process Inputs, Outputs (counts) Inputs, Outputs (counts) Outcomes Outcomes Formative Formative Summative Summative Cost/Benefit Cost/Benefit ROI... ROI...

9 How does OBE fit libraries? Inputs — How much we use Inputs — How much we use Outputs — How much we do Outputs — How much we do Performance quality — How well we do Performance quality — How well we do Economic value — What we’re worth Economic value — What we’re worth Outcomes — What good we do Outcomes — What good we do

10 How does OBE fit libraries? OBE changes focus, from activities to benefits OBE changes focus, from activities to benefits OBE needs to be part of the program design OBE needs to be part of the program design OBE shows to what extent a program met its own goals OBE shows to what extent a program met its own goals

11 Fears & Realities Will use too much time Will use too much time – Consider scope of the evaluation – Consider scope of the evaluation Visitor privacy may be compromised Visitor privacy may be compromised – Ask for volunteers – People like to tell you what they think

12 Fears & Realities Takes money from other priorities Takes money from other priorities – Funds are available – Results help leverage other funds Library impact cannot be measured – Show short-term, immediate impact Library impact cannot be measured – Show short-term, immediate impact

13 More Realities Libraries collect similar information to improve services and user satisfaction Libraries collect similar information to improve services and user satisfaction Evaluation can increase participation, improve services, and leverage funds Evaluation can increase participation, improve services, and leverage funds

14 Why measure outcomes? Know if program met purpose Know if program met purpose Improve programs Improve programs Guide management Guide management Communicate program impact Communicate program impact Satisfy funder’s need to know Satisfy funder’s need to know

15 What is a program? Series of services & activities that lead to a goal Series of services & activities that lead to a goal Has a definite beginning & end Has a definite beginning & end Meant to change attitude, behavior, Meant to change attitude, behavior, knowledge, skill, status, or condition

16 How to develop a program Identify a need Identify a need Need can be based on Need can be based on  Your experiences  Program partner experiences  Formal or informal research

17 How to develop a program Look at assumptions & verify needs Students need structure after school Students need structure after school Students need homework help Students need homework help Kids like computers Kids like computers Our library can provide safety, structure, computer skills, and help with homework skills Our library can provide safety, structure, computer skills, and help with homework skills

18 How to develop a program Identify solution A structured after-school program to provide kids with computer skills and homework help Identify desired results Kids will learn basic computer skills Kids will learn basic computer skills Kids will be information literate Kids will be information literate Kids will get better grades Kids will get better grades

19 How to develop a program Look at stakeholders Include individuals, agencies, funding sources, competition, community groups and national and state affiliations. They influence: Desired outcomes Desired outcomes How results are communicated How results are communicated Type and nature of services Type and nature of services Who program serves Who program serves

20 How to develop a program Look at audience Who is served by a program depends on several factors: Stakeholders Stakeholders Assumed need Assumed need Mission and resources Mission and resources

21 Example Provides a M-F after-school computer homework center Kids 8 to 12 in Springfield Kids have basic computer skills, are information literate, and get better grades Program Purpose Does what For whom For what outcome

22 Program Elements Inputs ServicesActivities Outputs Resources What a program uses Tasks What a program does Products Quantity of work or products

23 Outcome-based program Inputs Outcomes ServicesActivities Outputs Changes in knowledge, skills, abilities, attitudes, condition or life status

24 Measuring outcomes OBE is not formal research OBE is not formal research OBE shows contribution, not attribution OBE shows contribution, not attribution OBE shows what results program achieved OBE shows what results program achieved

25 Measuring outcomes Building a logic model Logic model is the evaluation plan Logic model is the evaluation plan Shows how all elements fit together Shows how all elements fit together Helps assure desired results Helps assure desired results

26 Outcomes Logic Model P ROGRAM InputsServicesActivitiesOutputs OutcomeIndicatorApplied toData sources Data intervals Target Intended impact Observable and measurable behaviors or conditions The population to be measured Sources of information about conditions being measured When data is collected The amount of impact desired

27 Identify program outcomes Immediate Immediate Intermediate Intermediate Long term Long term Outcomes

28 Hints: A program may have many outcomes A program may have many outcomes Pick a few important outcomes to measure Pick a few important outcomes to measure One significant outcome may be enough One significant outcome may be enough Outcomes

29 Outcome 1 Participants have basic computer skills Outcome 2 Participants are “information literate” Outcome 3 Participants complete homework Outcomes

30 Indicators  Observable, measurable, clearly identifiable  Unambiguous  Several may apply to each outcome

31 Indicators Outcome 1: Participants have basic computer skills Indicator : # and % of participants who can word- process one complete assignment without error

32 Indicators Does using a word processor indicate basic computer skills? A word processor is basic tool A word processor is basic tool Word processing requires specific skills Word processing requires specific skills If kids can produce a short assignment, they If kids can produce a short assignment, they probably have one basic skill set probably have one basic skill set

33 Logic Model Outcome 1: Kids have basic computer skills 75% Target After “basic” training session Data interval Completed assignment Data source All kids who finished “basic” training session # and % of kids who can word process 1 complete assignment without error Apply toIndicator

34 Reports What should reports say?  We wanted to do what  We did what  So what

35 Reports Summarize participant characteristics Summarize participant characteristics Summarize inputs, activities/services, outputs, and outcomes Summarize inputs, activities/services, outputs, and outcomes Respond to influencers’ need for information Respond to influencers’ need for information Compare data from program start or previous period Compare data from program start or previous period Interpret results and make recommendations Interpret results and make recommendations

36 Report Elements  What did we achieve for our target audience? Outcomes  How many units did we deliver?  To whom? ( audience characteristics ) Outputs  What did we do? Activities & Services  What did we use?  How much did we spend?  How much did we consume? Inputs

37 Options: Hire a consultant Benefits Result may be seen as unbiased Result may be seen as unbiased Professionals have most expertise Professionals have most expertise Process may be more efficient Process may be more efficient Offers outside perspective Offers outside perspective Who will do the work?

38 Options: Hire staff evaluator Benefits May reduce cost May reduce cost Greater understanding of your program Greater understanding of your program Greater commitment to the process Greater commitment to the process Who will do the work?

39 Options: Train existing staff (ideal) Benefits Integrate evaluation into routine management activities Integrate evaluation into routine management activities Staff know programs and audience Staff know programs and audience Skills transfer to all programs Skills transfer to all programs Who will do the work?

40 What do you get? Small investment: Numbers, audience characteristics, and customer satisfaction Small investment: Numbers, audience characteristics, and customer satisfaction Low to moderate investment: Immediate changes in knowledge, behaviors, and attitudes Low to moderate investment: Immediate changes in knowledge, behaviors, and attitudes

41 What do you get? Moderate to high investment: Attribute short-term changes in audience skills, knowledge, behaviors, and attitudes to program Moderate to high investment: Attribute short-term changes in audience skills, knowledge, behaviors, and attitudes to program High investment: Short- and long- term impact, attribution of impact to program, variables influencing impact High investment: Short- and long- term impact, attribution of impact to program, variables influencing impact

42 Your Action Plan Discuss benefits to your library Discuss benefits to your library Get more information Get more information Consider how to apply OBE Consider how to apply OBE Develop a plan Develop a plan

43 We have Talked about uses of OBE Talked about uses of OBE Talked about benefits Talked about benefits Defined terms Defined terms Described process Described process Discussed how results are used Discussed how results are used

44 Starting Places: Perspectives on Outcome-based Evaluation for Libraries and Museums (2001), IMLS Perspectives on Outcome-based Evaluation for Libraries and Museums (2001), IMLS IMLS bibliography IMLS bibliography

45 Starting Places: Action Plan for Outcomes Assessment in Your Library Peter Hernon and Robert E. Dugan Action Plan for Outcomes Assessment in Your Library, Peter Hernon and Robert E. Dugan (2002), ALA Editions A comprehensive plan specifically for librarians; provides data collection tools for measuring learning outcomes that link outcomes to user satisfaction W.K. Kellogg Foundation Evaluation Handbook (1998) W.K. Kellogg Foundation Evaluation Handbook (1998) Thorough introduction for organizations new to OBE

46 Karen Motylewski Institute of Museum and Library Services 1100 Pennsylvania Avenue, NW Washington, DC For more information