1. ANNUAL LECTURE Heterogeneous Agents, Social Interactions and Causal Inference This talk is based on “Heterogeneous Agents, Social Interactions, and Causal Inference” by Guanglei Hong and Stephen W. Raudenbush, to appear in Morgan, S. (Ed.) Handbook of Causal Analysis for Social Research (Springer 2012) and draws on two examples originally reported in: Savitz-Verbitsky, N. and Raudenbush, S.W. (in press). Evaluating community policing program in Chicago: A case study of causal inference in spatial settings. To appear in Epidemiologic Methods; and Raudenbush, S.W., Reardon, S. and Nomi, T. (in press). Statistical analysis for multi-site trials using instrumental variables. To appear in Journal of Research and Educational Effectiveness. The research reported here was supported by a grant from the Spencer Foundation entitled “Improving Research on Instruction: Models Designs, and Analytic Methods;” and a grant from the W.T. Grant Foundation entitled “Building Capacity for Evaluating Group-Level Interventions.” 21 ST MARCH 2012 Stephen W. Raudenbush Lewis-Sebring Distinguished Service Professor in the Department of Sociology at the University of Chicago and Chairman of the Committee on Education

2. Abstract This talk will focus on two pervasive features of social interventions designed to increase human health, skills, or productivity. First, the interventions are usually delivered by human agents – physicians, teachers, case workers, therapists, police officers, or workplace managers - who tend to be ‘heterogeneous’ in the sense that they differ in their beliefs, training, and experience. These agents enact the intervention and shape its effects. Second, the participants in these interventions – patients, pupils, employees or offenders - are typically clustered in organizational settings, and social interactions among these participants influence the success of the intervention. In this presentation, Stephen will argue that causal models conventionally used in medical research are not well suited to study these interventions. Instead, he proposes a model in which the heterogeneous agents and social interactions among participants shape participants’ response to an intervention. Stephen will illustrate this model with studies of community policing and high-school curricular reform.

3. Outline Counter-Factual Account of Causation The “drug-trial paradigm” for causal inference An alternative paradigm for social interventions Heterogeneous agents Social interactions among participants Examples Community policing High School Curricular Reform Conclusions

4. Counter-factual Account of Causality In statistics (Neyman, Rubin, Rosenbaum) In economics (Haavelmo, Roy, Heckman)

5. Potential Outcomes in a Drug Trial Y (1) : Outcome if the patient receives Z = 1 (the “new drug”) Y (0) : Outcome if the patient receives Z = 0 (the “standard treatment”) Y (1) – Y (0) : Patient-specific causal effect E (Y (1) – Y (0) ) =  : Average causal effect

6. Stable Unit Treatment Value Assumption (Rubin, 1986) Each patient has two potential outcomes Implies – Only one “version” of each treatment – No “interference between units” Implies the doctor and the other patients have no effect on the potential outcomes

7. Formally…

8. Failure of SUTVA in Education Teachers enact instruction in classrooms –Multiple “versions of the treatment” Treatment assignment of one’s peers affects one’s own potential outcomes –EG Grade Retention –Hong and Raudenbush, Educational Evaluation and Policy Analysis, 2005 –Hong and Raudenbush, Journal of the American Statistical Association, 2006

9. Group-Randomized Trials Potential outcome Thus, each child has only two potential outcomes – if we have “intact classrooms” – if we have “no interference between classrooms”

10. Limitations of cluster randomized trial Mechanisms operate within clusters * Example: 4Rs teachers vary in response classroom interactions spill over We may have interference between clusters * Example: community policing

11. Alternative Paradigm Treatment setting (Hong, 2004): A unique local environment for each treatment composed of * a set of agents who may implement an intervention and * a set of participants who may receive it Each participant possesses a single potential outcome within each possible treatment setting Causal effects are comparisons between these potential outcomes

12. Example 1: Community Policing (joint work with Natalya Verbitsky-Shavitz) Let Z j =1 if Neighborhood j gets community policing Let Z j =0 if not Under SUTVA

13. Relaxing SUTVA Potential outcome for any unit depends on the treatment assignment of ALL units in the population, Individual Causal Effect: Population Average Causal Effect:

14. “All or none” 1 1 1 1 1 0 0 0 0 0

15. “Shall we do it in my neighborhood?” 1 1 0 1 0 0 1 0 1 0

16. Do it only in high-crime areas: effect on those areas 1, HC 0, LC 1, HC 0, HC 0, LC 0, HC

17. Do it only in high-crime areas: effect on low-crime areas 1, HC 0, LC 1, HC 0, HC 0, LC 0, HC 0, LC

18. Spatial Causal Assumptions (1) Functional Form: 1, #3 1, #1 0, #4 0, #5 1, #2

19. Longitudinal Design: 25 districts, 279 “beats” 919293949596979899 No community policing 25 20 0000 Community policing 0005525

20. Results Having community policing was especially good if your surrounding neighbors had it Not having community policing was especially bad if your neighbors had it *** So targetting only high crime areas may fail***

21. Example 2: Double-dose Algebra Requires 9 th -graders to take Double-dose Algebra if they scored below 50 percentile on 8 th -grade math test 1200 students in 60 Chicago high schools

22. Double-dose Algebra enrollment rate by math percentile scores (city wide) Enrollment Rates ITBS percentile scores

23. Conventional Mediation Model (T, M,Y model) Cut off (T) Double-Dose Algebra (M) Algebra Learning (Y) Assume no direct effect of T on Y (exclusion restriction) Δ= Effect of double dose on the “compliers” Δ Γ= Effect of assignment to double dose (“ITT” effect) Nomi, T., & Allensworth, E. (2009) Γ Δ

24. Effects of Double-dose Algebra: District-wide average Effect of cutoff on taking DD (average compliance rate): Increase prob by.72 District-wide average ITT effect on Y: Average effect≈0.15 District-wide average Complier-Average Treatment Effect Average ≈0.21 double-dose algebra effects varied across schools

25. But the policy changed classroom composition!!

26. Classroom average skill levels by math percentile scores Pre-policy (2001-02 and 2002-03 cohorts) Post-policy (2003-04 and 2004-05 cohorts)

27. Implementation varied across schools in--- Complying with the policy Inducing classroom segregation

28. Exclusion Restriction Revised T-M-C-Y model Cut off (T) Double-Dose Algebra (M) Algebra score (Y) Classroom Peer ability (C)

29. Research Questions 1)What is the average effect of assignment to DD? (“ITT effect”) 2)What is the average effect of taking double-dose algebra? (effect “on the compliers”). 3)How much do these effects vary across schools? 4)What is the effect of taking double-dose Algebra, holding constant classroom peer ability? 5) What is the effect of classroom peer ability, holding constant taking double-dose Algebra?

30. Results Degree of sorting LowAverageHigh ITT0.210.150.11 Complier effects0.290.230.14 School N19 22 The effect of double-dose algebra on algebra scores by the degree of sorting We now estimate the effect of taking DDA and classroom peer composition

31. Statistical Models Stage 1: the effect of Cut-off on Double Dose and Peer Ability Stage 2: the effect of M and C on Y 31

32. Stage 1 Results: the average effect T on M and C Double-dose algebra enrollmentPeer composition Coeff 0.72 *** -0.24 *** SE 0.03 The effect of the cutoff score (T) on double-dose algebra enrollment (M) and peer composition (C) Note: *** p<.001, ** p<.01, * p<.05 32

33. Context specific effects: The effects of cutoff score on double-dose algebra enrollment and peer ability The effect of cut off score on peer ability The effect of cut off score on double-dose algebra enrollment

34. Stage 2 results: The effect of M and C on Y The average effect of taking double-dose algebra (M) and peer ability (C) on Algebra test scores Double-dose algebra enrollment Classroom Peer composition Coeff 0.30 *** 0.40 *** SE0.060.12 34

36. 5. Conclusions The reform enhanced math instruction for low- skill students, and that helped a lot The reform also intensified tracking and that hurt On balance the effect was positive, but much more so in schools that implemented double dose with minimal tracking

37. Final Thoughts Conventional causal paradigm: * a single potential outcome per participant under each treatment Alternative paradigm * a single potential outcome per participant in each treatment setting - aims to avoid bias -open up new questions Policy implications are potentially large