1. American Evaluation Association Annual Conference Friday October 17, 2014 Denver, CO Session #817

2.  Background  Demonstrate a simple procedure that might help you explain key concepts behind an RCT to a lay audience  Answer questions and hear your comments

3.  TEG has evaluated over 150 education-based programs over the last 20 years;  Most are multi-year programs;  Rigorous impact evaluations are increasingly required by many federal agencies (DOE SAMSHA, OJJDP);  DOE follows What Works Clearinghouse standards (WWC).

4.  RCTs and RDDs meet WWC evidence standards without reservations;  QEDs meet WWC evidence standards with reservations

5.  Terminology  Baseline equivalence?  Counterfactual?  Contrast effect?  Implementation fidelity?  Most are unfamiliar with or have forgotten Research Design 101  Abstract nature of the issue Demo, like a picture, is worth a 1000 words!

6. #1. Buy-in from program staff Without buy-in, run the risk of  Sabotage to design fidelity  Disgruntled staff poor implementation But what does buy-in mean?  Giving support willingly, via a shared understanding and agreement

7. So, if….  Staff support is a key ingredient to a successful RCT  Understanding is needed for staff support Q: How do you promote understanding in a non- threatening, easily digestible way to uninitiated or apprehensive staff? A: By way of simple analogy, using apples and oranges, and referencing the concepts described back to the program.

8. Context to this demonstration:  RCT impact evaluation of a school-based mentoring program serving at risk students  More students applied (312) than can be served (150).  Presentation of RCT to program staff

9. Materials:  Bag of apples (8-10)  Bag of oranges (8-10)  A few bananas (optional)  Large paper bag  Spray bottle with water Time:  20-30 minutes Objectives:  Impart an understanding of: ▪ Baseline equivalence; ▪ Counterfactual and contrast effect; ▪ Treatment fidelity

10. We’ve all heard the term You can’t compare apples and oranges, right? Why not? “In this bag I’ve got everyone who applied to our program (the few bananas over there applied but didn’t meet eligibility criteria). Although we can’t serve them all, we can do a rigorous test of the program. To do that, we will randomly assign our applicants to Tx and control groups.”  Have a staff member reach into the bag and select the first piece of fruit touched (apple or orange)  Alternate placing the selected fruit into either the Tx or control group;  Bananas symbolize non-eligible applicants.

11. Concept of Baseline Equivalence: 1. Randomization produces two equal groups on:  Known characteristics: On the outside they are same color, size, texture, etc.;  Unknown characteristics: they are likely the same on the inside (same color, # of seeds, etc.); Key point: Randomization cancels out individual differences and creates two equal groups. Program app: 150 TX, 162 control

12. Concept of Baseline Equivalence 2. Randomization is fair. No “creaming” the sample or selection bias.  “What happens if we select only perfect apples and oranges into the Tx group?”  Key point: All units have an equal opportunity to be in the Tx group  Program app: Randomization done independently via computer

13. Counterfactual and Contrast effect “Now that I’ve got 2 equal unbiased groups, I’m going to give the treatment group the intervention, which is designed to improve the taste of the fruit.”  Spray the treatment group fruit with the spray bottle 5-6 times  Do nothing to the fruit in the control group

14. Counterfactual 1. Given two equal groups, with one getting the Tx and one not, we are able to assess the impact of the intervention compared to no intervention.  Key point: without a control group, we would not be able to make causal attribution.  Program app: Can say with confidence that mentoring works.

15. Contrast effect 2. But do fruit farmers really do absolutely nothing to improve their fruit?  Key point: Probably not, so it’s important to monitor what the control group does get  Program app: do students in the control group get anything similar to mentoring?

16. Concept of Treatment Fidelity “Did anyone notice:  How many times I sprayed the fruit?  How close the sprayer was?  The quality of the water I used?”

17. Concept of Treatment Fidelity Evaluation TermDemonstration Application DosageWith what frequency, intensity, and duration did I deliver the spray? ParticipationDid all the fruit get sprayed? The same amount of spray? QualityWas it tap water or spring water? Clear or cloudy? Warm or cold? AdherenceWas it sprayed correctly? ResponsivenessDid the apples and oranges like getting sprayed? AdaptationDid I alter my spray technique?

18. Concept of Treatment Fidelity Evaluation TermProgram Application DosageWhat is the frequency, intensity, & duration of mentoring? ParticipationDid all students receive the same amount of mentoring? QualityWas the mentoring of high quality (mentors well trained? On time? Engaging?) AdherenceWas mentoring delivered as planned (1-1, at school, over lunch)? ResponsivenessDid students enjoy being mentored? AdaptationDid mentoring change in response to changing circumstance ?

19. Other adaptations:  To demonstrate outliers, include in the bag a green apple (or a rotten apple!)  To demonstrate multiple outcomes, can taste the fruit; grade its color; measure its size, etc.  To demonstrate overall and differential attrition, remove some fruit accordingly.  To demonstrate contamination and crossover, shift a few pieces of fruit from the control to treatment group.

20. Could this be applied to a QED? How might it be to applied to an RDD? How to illustrate differential outcomes? Other adaptations to describe design concepts?

21. Joel Philp Senior Director of Research The Evaluation Group 169 Laurelhurst Ave Columbia SC 29210 (803) 454-2006 Joel@evaluationgroup.com www.evaluationgroup.com