Step 2. Describe Program Logic Models & Outcome Measurement Luann D’Ambrosio, MEd

Exercise Improving on a hobby… Expectations by you and your agency

CDC Evaluation Framework

Overview Measuring and articulating what we do and what we accomplish Why? What does this process look like? What does it mean? How might I do this? What next?

Exercise When did you have to articulate what you do at work and why? To whom? How did it go?

Goals Understand the basics of performance/outcome measurement for our services Appreciate the importance of measurement systems for public health activity Goal 1- Build capacity in the local public health workforce for understanding the basics of performance measurement for public health services for children and families. Goal 2- Build consensus and appreciation within the local public health workforce on the importance of measurement systems for public health activity. …to improve quality and improve health! No EXPERTS expected out of this training No test in the end Possible “prizes” however

Objectives Identify components and uses of a logic model Describe outcome measures and why they are important Learning Objectives Describe the components and uses of a logic model Define outcome measures and why they are important Differentiate between indicators and outcome measures Describe some potential uses of outcome measures Identify measurable outcomes Determine levels (e.g. community, system, agency, program) of outcomes Define outcomes vs. goals & objectives

Outcome measurement is a systematic way to assess the extent to which a program has achieved its intended results. Outcome measurement is a systematic way to assess the extent to which a program has achieved its intended results. We have often (Yesterday). . .been more often focused on what we did. More and more (Today). . . We are focusing on what happens when we do it! To get there…. Outcome Measurement starts w/ the simple but IMPORTANT question: What is changed in the lives of our families, agency, community, system as a result of what we’ve been doing?? Our outcomes are thereby, less like: How many visits did I make How many people came to my clinic And more like: No. of cmty members that voted for a violence-free school zone No of pg women getting AP care in the 1st trimester Proportion of healthy foods snacks offered & purchased out of school vending machines Yesterday: More often focused on what we did. Today: Focusing on what happens when we do it!

Ways to Think About Outcome Measurement Performance measurement Evaluation Logic models We can think about Outcome measurement in terms of: Performance measurement Evaluation Logic models Other None of these are not mutually exclusive! interrelated The terms we will use are really moving targets Can mean something different to different persons They all advance the cause of measuring outcomes Performance/Outcome measurement can be simply understood to be regular collection & reporting of data to track work produced results achieved. Outcome measurement is not an end itself, but rather a means to an end. Outcome measurement is of little value unless it leads to further decisions and actions that seek to improve the performance of a program (or agency or individual) What outcomes are Important to achieve (are directly related to our core business) Meaningful (make a real difference for participants) Most useful (help in decision-making) Most reasonable (non-trivial, logical) Most realistic (achievable, given our resources) These are not mutually exclusive!

Why Do We Need to Do This? To provide accountability To improve program quality To make decisions about resource allocation To help programs market themselves To understand our contribution to our community’s health Why do Outcome Measurement? To provide accountability To improve program quality To support decision-making about resource allocation To help programs market themselves …to improve health! Background: Increasing emphasis on articulating what we do in PH and what our contributions are to the public’s health. R/t: Funding pressures The outcomes of our programs (accountability, programs marketing) The contribution of our resources/staff/PHNs to the health of communities Supporting recruitment/retention Staffing resources (accountability, resource allocation) Setting standards for the public to expect from us and request from their policy makers (quality, accountability) A recognition that we are not all prepared yet to do this HRSA funding for training to the PH practice community WA Family Child Health program directors are doing work around this already—creating logic models To evaluate and improve programs To communicate our successes Telling our story of how we did it and what happened as a result The “so what” of our efforts

“Everyday” Example: Marathon Runner Goal: To do better than last year’s time Resource: New running shoes Activities: Running workouts Outputs: Train 5x per week; run increasing #s of miles Outcome: Training times/mile, overall time/distance, energy level simple example that we can relate to at one level or another (running or watching others run) an example that allows us to readily identify some of the important concepts we associate with the term performance management In this example-- easy to determine the who, equally apparent should be what this person is seeking to accomplish. Even if not runners, it should be clear that training at various distances keeping track of time for these distances are useful activities that can assist a serious runner in determining whether some change in their race strategy might make sense in order to maximize their outcome and improve on last year’s time in this event.

How Do We Do This? Set a goal Define activities Identify inputs or resources Define outcomes Develop outcome measures Build consensus around these measures Continue the runner example here…. Set a goal. (Run a faster marathon.) Define activities. (Train 5x a week and increase # of miles.) Identify inputs or resources. (Purchase new shoes.) Define outcomes. (Develop physical conditioning and increase running skills.) Develop outcome measures or indicators. (Improve running times and endurance.) Build consensus around these measures. (Important—if I’m doing this w/ my friend.) Resources Activities Outputs Outcomes Goals

Logic Models: “So That” Chains Resources Activities Outputs Outcomes Goals so that so that so that so that Logic Model So why do we need those new running shoes…. So that….we can go running So why do we go running So that….increase our mileage, improve stamina So why do we want to increase our miles So that….we can improve our time from last year Or decrease our blood pressure In the past we often stopped asking the “so that” questions after our INPUTS This kept us from answering the real questions related to what we intended to ACHIEVE These elements are what make up a LOGIC MODEL

Logic Models: Public Health Example RESOURCES ACTIVITIES OUTPUTS OUTCOMES GOALS Staff Immunization clinic Shots Less VPD Healthier communities OFS, page 29 Resources— program inputs—the elements that constitute our program. e.g. staff, funds, participants What must this program have in order to function well? Activities— methods for providing our programs to participants specific processes and/or events which you undertake to carry out the program E.g. training, advocacy, case mgmt What must we do to achieve our intended results? Outputs— Services or product units E.g. the # of clients we served How much do you do for how many people? Outcomes— The changes in people’s lives (or our community or our agency) that you believe will happen as a result of your efforts So what? What difference does our program make? Should be Important to achieve (directly related to our core business) Meaningful (make a real difference for participants) Useful (help in decision-making) Reasonable (non-trivial, logical) Realistic (achievable, given our resources) Goals— ultimate impact our program is expected to make Beyond what you or your program do alone the core mission/value does my program addresses? HAVE A LUNCH BREAK HERE????? so that so that so that so that

Exercise Create a So That chain for your project or program.

Outcome Evaluation Logic Model Resources What must this program have in order to function well? Goals What core community value does my program address? Activities What must we do to achieve our intended results? If we’re planning we might go backwards around this diagram If we’re evaluating we will go forwards Define a common goal Define common outcomes Define our outputs units of service Define common activities Interventions to improve individual and family life course Identify the resources we have or need to achieve this Outcomes So what? What difference does our program make? Outputs How much do you do for how many people?

Developing Logic Models Process Outcomes RESOURCES ACTIVITIES OUTPUTS OUTCOMES GOALS Program inputs. Elements or ingredients that constitute the program. Methods for providing the program. Specific processes or events undertaken. Units of service or product units. How many, over what duration? Short, intermediate, or longer-term changes anticipated in participants’ lives or in organizational or community conditions. Ultimate impact(s) expected to occur, usually beyond what one program alone can achieve. Define a common goal Define common outcomes Define our outputs Define common activities Service-based e.g. WIC vouchers School-based clinics Immunizations CD case investigation Dental sealants Family planning Education-based Breastfdg ed & support Car seat education Nutrition education Tob. prevention Child Care Consultants Mobilization-based Train cmty mbrs in a body of info Assm’t & data sharing Coalition bldg Identify/build relationships w/ diverse stakeholders Educate comty & policy-makers re scope of situation Identify the resources we have or need to achieve this DO PH RAP SONG HERE…..???

Why a Logic Model? A logic model Clarifies each program element Shows the relationship of resources and activities to expected outcomes Summarizes how program parts relate to the whole Makes explicit the theory of how the program works Identifies categories to measure in the program evaluation

Exercise Start your own (work related) logic model (individually)

Logic Models: “So That” Chains Resources Activities Outputs Outcomes Goals so that so that so that so that Logic Model So why do we need those new running shoes…. So that….we can go running So why do we go running So that….increase our mileage, improve stamina So why do we want to increase our miles So that….we can improve our time from last year Or decrease our blood pressure In the past we often stopped asking the “so that” questions after our INPUTS This kept us from answering the real questions related to what we intended to ACHIEVE These elements are what make up a LOGIC MODEL

Exercise Logic model scramble (everyone)

Outcome Levels Individual performance outcomes Program outcomes Agency outcomes System outcomes Community outcomes OFS, Page 23 1) Individual performance outcomes 2) Program outcomes Changes in individs, families, organizations, or the cmty that can be influenced by the program alone 3) Agency outcomes Community level change based on the agencies as a GROUP of programs e.g. increased birth spacing (our FP clinic, PHN HVs, DSHS liaison, etc) 4) System outcomes A group of agencies working together to increase safety for vicitms of DV 5) Community outcomes A joint effort of multiple agencies, funders, community groups Established through community goal setting E.g. decreasing teen pg

Developing Logic Models Process Outcomes RESOURCES ACTIVITIES OUTPUTS OUTCOMES GOALS Program inputs. Elements or ingredients that constitute the program. Methods for providing the program. Specific processes or events undertaken. Units of service or product units. How many, over what duration? Short, intermediate, or longer-term changes anticipated in participants’ lives or in organizational or community conditions. Ultimate impact(s) expected to occur, usually beyond what one program alone can achieve. Define a common goal Define common outcomes Define our outputs Define common activities Service-based e.g. WIC vouchers School-based clinics Immunizations CD case investigation Dental sealants Family planning Education-based Breastfdg ed & support Car seat education Nutrition education Tob. prevention Child Care Consultants Mobilization-based Train cmty mbrs in a body of info Assm’t & data sharing Coalition bldg Identify/build relationships w/ diverse stakeholders Educate comty & policy-makers re scope of situation Identify the resources we have or need to achieve this DO PH RAP SONG HERE…..???

Exercise Further develop your own (work related) logic model (individually)

Reading Logic Models 1 2 3 4 5 Your planned work Your intended results Certain resources are needed to operate your Program. If you have access to them, then you can use them to accomplish your planned Activities. If you accomplish your planned activities, then you will hopefully deliver the amount of product or service that you intended. If you accomplish your planned activities to the extent you intended, then your participants will benefit in certain ways. If these benefits to participants are achieved, then certain changes in organizations, communities, or systems might be expected to occur. Resources/Inputs Activities Outputs Outcomes Impact 1 2 3 4 5 Your planned work Your intended results

What Next With Outcome Measures? Remember… outcome measurement is not an end itself, but rather a means to an end. Outcome measurement is of little value unless it leads to further decisions and actions that seek to improved program (or agency or individual) performance. Performance improvement doesn’t miraculously happen. There are a number of specific steps and techniques These steps are used in and outside of the health field to improve quality and performance.