1. Field Guide to Living a Data-Rich Life Webinar 3: Top-down and Bottom-up Approaches to Designing Outcomes Hosted by Durham’s Partnership for Children, with support from the North Carolina Partnership for Children Presented by Compass Evaluation and Research

2. Welcome Webinar 3 in our year long data quality series Last time (webinar 2), we discussed logic models, the Theory of Change, and how to use these models to create an evaluation plan Today, we will: Review concepts from Webinar 2 Discuss five top-down and bottom-up practical strategies for moving from understanding the need to designing outcomes Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

3. Recap from Webinar 2 How do we start to identify outcome ideas? Logic Models Theory of Change Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

4. Sample Logic Model Template Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Inputs Strategies & Activities Outputs Short-Term Outcomes Longer-Term Outcomes Tips: Existing resources, guidance, supports, etc. that can be provided to the program Target Population Tips: Ensure the target population is consistent with the need Tips: Account for the majority of your program’s expenditures Tips: Identify how many of the target population you will serve How many services you will provide Other deliverables Tips: What changes can occur within one year? Tips: To what extent have you responded to the need? Need Tips: The need to which your program is responding

5. Sample Logic Model: How will the program operate? Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Inputs Strategies & Activities Outputs Short-Term Outcomes Longer-Term Outcomes Administrative and bookkeeping support Advisory Council Target Population Young families within one year of completing High School or in first year of college Home visitation services # of clients served with home visits; total # of home visits Number and % of clients completing their one-year educational goals Number and # of clients completing a college degree Group workshops Advocacy services Facilitated referrals Education support # of clients attending group workshops; total # of group workshops # of clients receiving advocacy services; total # and type of advocacy services provided # of clients receiving referrals; total # and type of referrals provided # of clients receiving education support; total # and type of education support services provided Number and % of clients improving parenting skills Number and # of clients exhibiting family resilience Need Without support and resources, young adults often fail to complete their education, creating a risk for economic stability

6. Top-Down versus Bottom-Up Approaches to Designing Outcomes Use both! Top-down: Understanding the funder’s goals and objectives Bottom-up: Understanding your community or target population Your logic model, Theory of Change, and outcome development benefit from both perspectives Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

7. Practical Strategies Step 1 (Top-Down): identify the “Big Picture Need”—the need domains the funder is interested in Step 2 (Top-Down): identify data that express the level of “Big Picture Need” in your community—typically quantitative Step 3 (Bottom-Up): understand HOW and WHY the need exists in your community or target population Step 4 (Bottom-Up): choose programs or models that will meet your community or population preferences or service needs Step 5 (Top-Down and Bottom-Up): design outcomes that are SMART Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

8. Step 1: What is the “Big Picture”? Why did the funder choose certain funding areas? What is the “public good” that is being addressed through funding? Public good: an investment that is of benefit to society-at-large Consider whether or not this Big Picture Need also is of concern in your community or for your target population You’ll see these ideas appear again in your LONGER- TERM outcomes Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

9. Step 2: Describe the Big Picture for Your Community Publicly-available data North Carolina Public Schools North Carolina State Center for Health Statistics Census.gov Department of Social Services And many more… Partner-supplied data Local branches of these state agencies Community partners who may have data Data collected for your community Focus on the quantitative Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research These same data may help you track your longer-term outcomes

10. Step 3: Understand HOW and WHY Collect data to understand how and why the need exists in your community or target population Focus groups, interviews, and surveys Community- or population-focused needs assessments Ask questions such as: Why does the need exist? What will it take to resolve the need? Does the community already have good programs—but in insufficient quantity? Does the community need different types of programs? Does there need to be a change in how programs are provided, such as days, times, locations, etc.? Are programs affordable and accessible? You’ll see these ideas again in your SHORTER-TERM outcomes and tracking of program implementation Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

11. Step 4: Choose a Program or Model that is a Good Fit Examine research and evidence for programs to see if they have been successful for your target population Compendia such as Smart Start’s Guide to Evidence-Based and Evidence-Informed Programs Google Scholar Academic search engines Clearinghouse ratings What Works Clearinghouse California Clearinghouse Promising Practices Network Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

12. Step 5: Design Outcome Measures that are SMART Well-defined need statements lead to high quality outcomes This is to say, needs and outcomes should be SPECIFIC Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Non-SpecificMore Specific Big Picture Need Statement Children are not arriving at school ready to learn. 30% of entering kindergarten students are below expectations in one or more developmental domains. Long-Term Outcome Measure Children arrive at school ready to learn. The number/percent of entering kindergarten students who start kindergarten at or above developmental expectations in all domains.

13. Step 5: Design Outcome Measures that are SMART High quality outcome measures are MEASURABLE. Will high-quality, outcome-specific data be available for the evaluation? If not, can you capture these data through your evaluation, with the available time and resources? Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

14. Step 5: Design Outcome Measures that are SMART Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Sample Long-Term OutcomePoor Data MatchGood Data Match The number or percent of entering kindergarten students who start kindergarten at or above developmental expectations in all domains. For districts served by the program, over time: - Number of entering kindergarteners - Number of kindergarteners considered ready by kindergarten teachers For districts served by the program, over time: - Number of entering kindergarteners - Number of entering kindergarteners who received developmental assessments (for one or more developmental domains) Of the students who received assessments: -the number and percent who were assessed as at or above developmental expectations, for each domain assessed.

15. Step 5: Design Outcome Measures that are SMART Good outcomes reflect changes that are ACHIEVABLE within a given timeframe. Look to the research to determine: What financial, tangible, and other resources will be necessary to achieve the goal? What is the likelihood that each program participant will experience success in the program? Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

16. Step 5: Design Outcome Measures that are SMART RELEVANT outcomes address the degree to which the underlying need or problem has been alleviated Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Short-Term Outcomes Longer-Term Outcomes Is the program achieving the earliest predictors of success? Is the program operating in a way that demonstrates a good fit with the population? The extent to which need has been addressed Need Quantitative expression of the “big picture” need Quantitative and qualitative expression of HOW and WHY the need exists Informed by program design—does the program model meet your population’s service patterns and preferences? Informed by research— does the evidence suggest your program will resolve the need for your target population?

17. Step 5: Design Outcome Measures that are SMART Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Sample OutcomeShort TermLong Term The number or percent of entering kindergarten students who start kindergarten at or above developmental expectations in all domains. Number and percent of children identified as needing support in the birth to prekindergarten years Number and percent of children effectively served through accessible and affordable programs --children receive services relevant to their individual needs For districts served by the program, over time: - Number of entering kindergarteners - Number of entering kindergarteners who received developmental assessments (for one or more developmental domains) Of the students who received assessments: -the number and percent who were assessed as at or above developmental expectations, for each domain assessed. Will your choice of program model work with your target population to produce these short-term outcomes? Is your program model proven successful at achieving this longer- term outcome?

18. Step 5: Design Outcome Measures that are SMART Good outcomes are achievable within a defined period of TIME. Bottom-Up: Short-term outcomes Reflect the HOW and WHY of responding to the need Provide earliest evidence the program model is working Top-Down: Longer-term outcomes Reflect the need the funder is trying to address May require two or more years to achieve Depend upon successful completion of short-term outcomes If you don’t identify HOW and WHY a program might be successful with your population, you run the risk of failing to achieve your long-term program goals Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

19. School Readiness Example Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Affordable and accessible programs are available, that identify to individual child needs There is an increase in the # of % of students who start kindergarten at or above developmental expectations in all domains. TOP-DOWN: 30% of entering kindergarten students are below expectations in one or more developmental domains. BOTTOM-UP: Too few community members have access to and use affordable programs that respond to child needs. Affordable and accessible programs are available, that respond to individual child needs Parents have access to and can afford fully utilize the available programs Children exhibit developmental gains Program investments Fidelity of implementation measures Short and long term outcome measures Big Picture Need Why the need exists in your community

20. Cheat Sheet for Designing Outcomes What is the Big Picture Need? How and why does the need exist in my community? What will it take to resolve the need? What are the first signs the program is working? How will I know if the need is being addressed? Tip: Reference the funder’s goals and funding interests Tip: Collect data in your community to understand why the need exists locally Tip: Ask your target population the specific nature and level of services they will respond to Tip: Identify a research-supported program or programs that have proven successful for a similar community or target population Tip: plan to capture evaluation data that show the program is being implemented as designed Tip: plan to capture short-term outcomes that show the target population is responding to the program Tip: capture data that use the same language and statistics as the originating need Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

21. Questions? Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research

22. Next Steps Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research The webinar series is designed to help you create a plan to capture, manage, analyze, and use high quality data: April 26, 2016: Integrating Quantitative and Qualitative Data into your Evaluation. May 25, 2016: Designing or Choosing Instruments: Surveys. June 28, 2016: Designing or Choosing Instruments: Observations. July 12, 2016: Designing or Choosing Instruments: Standardized Assessments. July 27, 2016: Choosing and Using Sampling in Your Evaluation. August 30, 2016: Best Practices in Data Collection and Management. September 28, 2016: Finding the Value in Evaluation: Cultural Relativity and Bias. October 25, 2016: Using Data: Effective Reporting and Grant Writing. November 30, 2016: Thinking Beyond Your Program: Evaluating Systems and Collaborations.

23. Additional questions? Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Megan Gray Durham’s Partnership for Children Phone: 919-403-6960 (extension 231) Email: megan@dpfc.netmegan@dpfc.net

24. Additional questions? Designing Outcomes Field Guide to Living a Data-Rich Life Compass Evaluation and Research Feel free to contact me, too! Sarah Heinemeier Compass Evaluation and Research Phone: 919-308-5019 Email: sarahhei@compasseval.com