1. Analysis for Designs with Assignment of Both Clusters and Individuals
Cristofer Price
SREE - March 2017

2. The Problem How to analyze data from designs where:
Part of sample has assignment of clusters
e.g. random assignment (RA) of districts, schools, classes
Part of sample has assignment of individuals within clusters
e.g., RA of schools within districts
e.g., RA of students within schools

3. Example Special education intervention in rural schools
In some schools, there are enough eligible students to fill treatment slots and also have students in the business-as-usual control condition
So there is within-school assignment
In other schools, there are not enough eligible students to fill the slots
So there is between-school assignment (T schools and C schools)

4. Prototypical Design with RA of Schools and RA of Students within Schools

5. Desired Outcome of Analysis Approach
Produce a point estimate and standard error of the average treatment effect where
Effect is averaged across both RA of schools and RA of students within schools designs
Impact estimate is unbiased
Impact standard error is unbiased

6. This Presentation is NOT About
Partially nested RCTs
Individuals randomized, but in T group individuals are placed into clusters, C group individuals are not
Roberts & Roberts (2005); Bauer et al (2008; Baldwin et al. (2011) Lohr et al., (2014); Sterba et al. (2014)
Individually randomized grouped treatment designs
Inability to get SE.s exactly right in individual RA designs where T and C individuals are placed into clusters (post- RA), and similarity of experiences within clusters induces correlation within clusters
Weiss et al. (2015); Lee & Thompson (2005)

7. This Presentation IS About
Two approaches to analyzing the mixed assignment design to produce a point estimate and standard error of the impact
Approach 1:
Perform two separate analyses, get estimates and SE.s, calculate a weighted average of the two estimates
Approach 2:
Analyze all data in a single multilevel model that has two separate sets of intercept terms
One set of random intercept terms for schools in the school RA design
One set of intercept terms for schools in the RA of students within schools design

8. Take Away Points I show how to implement the two approaches
I show how I assessed the extent to which they produced unbiased impact estimates and standard errors
I conclude that both approaches:
Work equally well to each other
Work about as well as standard approaches to analyzing the two samples separately
Produce more precise estimates than those obtained in separate analyses

9. Approach 1: Model For Sample with RA of Schools
Two level model
with students (level-1)
nested in schools (level-2)

10. Approach 1: Model For Sample with RA of Schools
Two level model
with students (level-1)
nested in schools (level-2)
And written in combined
form

11. Approach 1: Model For Sample with RA of Schools
(For simplicity, I’m showing models with no covariates)

12. Approach 1: Model For Sample with RA of Schools

13. Approach 1: Model For Sample with RA of Schools

14. Approach 1: Model For Sample with RA of Schools

15. Approach 1: Model For Sample with RA of Schools

16. Approach 1: Model for Sample with RA of Students within Schools
I’m going to describe three commonly used models for designs with random assignment of students within schools

17. Approach 1: Model for Sample with RA of Students within Schools

18. Approach 1: Model for Sample with RA of Students within Schools
Fixed Treatment effect

19. Approach 1: Model for Sample with RA of Students within Schools
Fixed Treatment effect
Fixed dummies for schools

20. Approach 1: Model for Sample with RA of Students within Schools

21. Approach 1: Model for Sample with RA of Students within Schools
Fixed dummies for schools

22. Approach 1: Model for Sample with RA of Students within Schools
Fixed dummies for schools
Treatment by school interaction terms.

23. Approach 1: Model for Sample with RA of Students within Schools
Fixed dummies for schools
Treatment by school interaction terms.
This model will produce a separate impact estimate for each school.

24. Approach 1: Model for Sample with RA of Students within Schools
Fixed dummies for schools
Treatment by school interaction terms.
This model will produce a separate impact estimate for each school. To get an overall estimate, need to calculate an average of the individual estimates (perhaps a weighted average)

25. Approach 1: Model for Sample with RA of Students within Schools

26. Approach 1: Model for Sample with RA of Students within Schools
Random Intercepts for Schools

27. Approach 1: Model for Sample with RA of Students within Schools
Random Intercepts for Schools
Random treatment effects

28. Approach 1: Get the Relevant Estimates from Models 1 and 2

29. Approach 1: Get the Relevant Estimates from Models 1 and 2

30. Approach 1: Create Weights
Create weights that are proportional to the inverse of the variance of the estimates

31. Approach 1: Calculate the Combined Treatment Effect Estimate

32. Approach 1: Calculate the Combined Treatment Effect Estimate
I call this the “meta-analytic” approach

33. Approach 1: Calculate the Combined Standard Error

34. Approach 2: Analysis of Both Samples in a Single Model
I will show:
Model 3: A single model that produces separate estimates for the two samples
This is to facilitate comparisons of the single model results to analyzing the data from the two sample separately
Model 4a: A combined model that produces a single combined fixed treatment estimate
Model 4b: A combined model that has a random treatment effect for the sample where students were randomized within schools

35. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students

36. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Random intercept terms for part of sample where schools were RA’d
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students

37. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Random intercept terms for part of sample where schools were RA’d
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students
Fixed intercept terms for part of sample where students were RA’d

38. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Random intercept terms for part of sample where schools were RA’d
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students
Fixed intercept terms for part of sample where students were RA’d
Treatment effect for part of sample where students were RA’d

39. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Random intercept terms for part of sample where schools were RA’d
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students
Fixed intercept terms for part of sample where students were RA’d
Treatment effect for part of sample where students were RA’d
=1 of Schools RA’d
=0 of Students RA’d

40. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Random intercept terms for part of sample where schools were RA’d
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students
Fixed intercept terms for part of sample where students were RA’d
Treatment effect for part of sample where students were RA’d
Difference between treatment effect for part of sample where schools where RA’d and part of sample were students were RA’d
=1 of Schools RA’d
=0 of Students RA’d

41. Fit a Single Model to Both Samples Get Two Separate Treatment Estimates
Random intercept terms for part of sample where schools were RA’d
Combined model for full sample with separate treatment estimates for subsets with RA of schools and RA of Students
I’ll show you that this model produces the same estimates and SE’s as Models 1 and 2a
Fixed intercept terms for part of sample where students were RA’d
Treatment effect for part of sample where students were RA’d
Difference between treatment effect for part of sample where schools where RA’d and part of sample were students were RA’d
=1 of Schools RA’d
=0 of Students RA’d

42. Approach 2: Obtain a Combined Estimate from a Single Model

43. Approach 2: Obtain a Combined Estimate from a Single Model
Fixed treatment effect

44. Approach 2: Obtain a Combined Estimate from a Single Model
Fixed treatment effect
Fixed dummies (intercepts) for schools
where students were RA’d

45. Approach 2: Obtain a Combined Estimate from a Single Model
Fixed treatment effect
Random intercepts for schools where schools were RA’d
Fixed dummies (intercepts) for schools
where students were RA’d

46. Approach 2: Using Software to Fit Model 4a
e.g., in SAS:

47. Approach 2: Using Software to Fit Model 4a
e.g., in SAS:
Two Set of school IDs

48. Approach 2: Using Software to Fit Model 4a
e.g., in SAS:
Two Set of school IDs
“class” statement creates dummy variables from the IDs

49. Approach 2: Using Software to Fit Model 4a
e.g., in SAS:
Two Set of school IDs
“class” statement creates dummy variables from the IDs
Fixed dummies (intercepts) for schools
where students were RA’d

50. Approach 2: Using Software to Fit Model 4a
e.g., in SAS:
Two Set of school IDs
“class” statement creates dummy variables from the IDs
Fixed dummies (intercepts) for schools
where students were RA’d
Random intercepts for schools where schools were RA’d

51. Approach 2: How to Code Data to Get the Two Sets of Intercepts

52. Approach 2: How to Code Data to Get the Two Sets of Intercepts
Here are the original
school IDs

53. Approach 2: How to Code Data to Get the Two Sets of Intercepts
Create two new sets of school IDs

54. Approach 2: How to Code Data to Get the Two Sets of Intercepts
Create two new sets of school IDs

55. Approach 2: How to Code Data to Get the Two Sets of Intercepts
Create two new sets of school IDs

56. Approach 2: How to Code Data to Get the Two Sets of Intercepts
For the part of the sample were schools were RA’d to T & C

57. Approach 2: How to Code Data to Get the Two Sets of Intercepts
For the part of the sample were schools were RA’d to T & C
Assign the original IDs to “Schid1”

58. Approach 2: How to Code Data to Get the Two Sets of Intercepts
For the part of the sample were schools were RA’d to T & C
Assign the original IDs to “Schid1”
And assign zeros to “Schid2”

59. Approach 2: How to Code Data to Get the Two Sets of Random Intercepts
For the part of the sample were students were RA’d to T & C
within schools

60. Approach 2: How to Code Data to Get the Two Sets of Random Intercepts
For the part of the sample were students were RA’d to T & C
within schools
Assign the original IDs to “Schid2”

61. Approach 2: How to Code Data to Get the Two Sets of Intercepts
For the part of the sample were students were RA’d to T & C
within schools
And assign zeros to “Schid1”
Assign the original IDs to “Schid2”

62. Approach 2: Obtain a Combined Estimate from a Single Model
Model 4b

63. Approach 2: Obtain a Combined Estimate from a Single Model
Model 4b
Has random treatment effects for the part of the sample where students were RA’d within schools

64. Approach 2: Obtain a Combined Estimate from a Single Model
Model 4b
Has random treatment effects for the part of the sample where students were RA’d within schools
And two sets of
random intercepts

65. Approach 2: Obtain a Combined Estimate from a Single Model
Model 4b
Has random treatment effects for the part of the sample where students were RA’d within schools
And two sets of
random intercepts
= 1 if schools RA’d
= 0 if students RA’d

66. Approach 2: Obtain a Combined Estimate from a Single Model
Model 4b
Has random treatment effects for the part of the sample where students were RA’d within schools
And two sets of
random intercepts
= 1 if schools RA’d
= 0 if students RA’d
Average
treatment
effect

67. How do these Approaches Perform?
Used simulated data to assess the extent to which each approach produces treatment effect estimates and standard errors that are unbiased. Three sets of simulations (10,000 replications each):
Set 1: Fixed set of schools and students. In each replication, schools and students assigned to different conditions (T or C)
Set 2: Fixed set of schools, but different students within each school in each replication. Distribution of student’s underlying abilities within each school stays the same across replications
Set 3: Infinite population of schools. In each replication, different schools and students selected, randomly assigned.

68. Simulated Data In each simulation:
True average treatment effect = 0.20
1/5 of schools or students true effect = 0.10
1/5 of schools or students true effect = 0.15
1/5 of schools or students true effect = 0.20
1/5 of schools or students true effect = 0.25
1/5 of schools or students true effect = 0.30
ICC = 0.20

69. Mean of impact estimates over the 10,000 simulations.
True impact is 0.20

70. Mean of standard error estimates over the 10,000 simulations.

71. True SE = square root of variance of impact estimates over the 10,000 simulations

72. Difference between mean of SE estimates and True SEs

73. Difference between mean of SE estimates and True SEs as a percent of the True SE

74. Same

75. Same

76. Same

77. The models that produce a single estimate from the two samples

78. The models that produce a single estimate from the two samples are more precise…

79. The models that produce a single estimate from the two samples are more precise than the models that produce separate impact estimates in each sample

80. But the extent to which there is bias in the SEs is about the same

81. Take Away Points
I presented two easy-to-implement approaches to analyzing data from a mixed assignment design
I showed how I assessed the extent to which they produced unbiased impact estimates and standard errors
I concluded that both approaches:
Work equally well to each other
Work about as well as standard approaches to analyzing the two samples separately
Produce more precise estimates than those obtained in separate analyses

82. Which to Use Either seems good
Meta-analysis approach would estimate different relationships between covariates and outcomes for the two samples
Single model combined approach would have common covariate effects across the two samples
(unless the covariates were interacted with the “SchoolAssign” dummy variable)

83. Thank you!