Measuring Success & Impact: Challenges & Opportunities for STEM Early Learning Programs Tiffany R. Lee University of Colorado Boulder University of Washington

Introductions Name, organization, role Short project description Experience with measuring success and impact (evaluation) – for this project or others Lee  2014

Words (wordle?) Design Implement Explore Evaluation Assessment Goals Measure Impact Success Refine Test Scale Learning Reach Data Change Outcomes Feedback Others? Lee  2014 Others: Evidence Scope Analysis Time

What outcomes would indicate to you that your program is achieving success?

First, let’s think about these questions.. What resources go into your program? (ex: funding, staff, partners) What activities does your program develop, support, and implement? What is produced from the activities and the program’s efforts? Lee  2014

What outcomes would indicate to you that your program is achieving success? Development of early childhood STEM lessons / activities? Increased effectiveness of program facilitators? Child learning of particular content or skills? Change in parents’ / caregivers’ behavior? Increased community reach? Lee  2014 What are the changes or performances that result from the program? Be specific.

Outcomes/Goals Inputs Activities Outputs What resources go into your program? (ex: funding, staff, partners) What activities does your program develop, support, and implement? What is produced from the activities and the program’s efforts? What are the changes or performances that result from the program? Program Logic Model

Some Considerations for Measuring Success…

Some Questions to Consider About Your Participants Does your program have repeat vs. one-time interactions over time with participants – children and/or caregivers? Do you have a way to contact and follow up with your program participants? About Your Program Do you have the same facilitators leading lessons / activities throughout the duration of your program? Does your program implement a lesson / activity once, or do you refine it over multiple iterations? Lee  2014 Your answers to these questions make a difference in what (and how) you can measure success!

Questions About Your Participants Repeat vs. one-time interactions over time with children and/or caregivers Can you measure changes in child / caregiver knowledge and skills? Alternatives: Changes in group behavior, facilitation, work products (ex: child artwork ) Lee  2014

Questions About Your Participants Participant contact info Can you collect information about what the child / caregiver does that is related to a program experience? Alternatives: Exit tickets, repeat interactions? Lee  2014

Questions About Your Program Same facilitators leading lessons / activity Can you measure changes in individual’s facilitation skills? Alternatives: Differences between facilitators Lee  2014

Questions About Your Program One-time vs. multiple iterations of lesson / activity What is the rationale for making changes to a lesson / activity? Do they result in changes in participant engagement (as a group)? Alternatives: Design principles Lee  2014

Methods for Measuring Success Methods Descriptive statistics Time sampling Event sampling Field notes Exit tickets Work samples Photographs Video / audio recordings Conversations with caregivers Team reflections Lee  2014 Considerations Time Timing / frequency Staffing Permissions Materials Subjectivity / Objectivity

Refining Outcomes and Identifying Measures of Success

Identifying Measures of Success Review Outcomes / Goals –Can you measure these given the constraints of your program? (Think about your access to participants and program design) –What does it look like when your program goals are met? Brainstorm possible measures –What resources will be needed? (additional staff member on site, develop protocols) –What are some challenges and considerations? Lee  2014

Indicators Methods/Measures Inputs What methods/methods will be used to measure your outcomes/goals? What resources will be needed to implement your methods/measures? What does it look like when your program goals are met? Measuring Success Challenges/Consideratio ns What are some anticipated challenges and considerations for implementing these measures?

Exit Ticket 1.What do you think will be the biggest challenge when implementing an evaluation plan? 2.What is most exciting to you about evaluation? 3.What is the most helpful thing you learned today? Lee  2014 Thank you! Tiffany Lee