1. Analysis of Experiments
Larry V. Hedges
Northwestern University
Prepared for the IES Summer Research Training Institute July 7 – 17, 2008

2. Software
Many kinds of software can provide valid analyses of experiments with complex designs
SAS Proc Mixed
STATA XTMixed
SPSS GLM
HLM
We will focus on HLM because it is very general and easy to use, but if you want to use SAS Proc Mixed, we will try to accommodate you in using that here
The choice of programs should be based on your level of comfort

3. What is HLM?
HLM is not a complete statistical package like SAS, SPSS, or STATA
HLM is a software tool that works with other statistical packages such as SPSS, SAS, or even EXCEL
Data management and preparation (recoding and creation of variables) is done in the general package (e.g., SPSS)
HLM then takes the data intended for analysis and performs various complex analyses on the data

4. How Does HLM Work?
HLM reads your data and creates summary files from it Multivariate Data Matrix (or MDM) files
There is one MDM file for each level of the data analysis. It essentially is the information necessary to carry out the analyses at each level of the HLM model
Every variable that you plan to use in the HLM analysis must be included in the MDM file at the level you intend to use it. If you decide to include new variables, you must create new MDM files
Your data must be sorted by the ID variable that links different levels. If it is not, you will get an error message

5. Specifying Analyses Know the design
Think through the levels of the design that will be included in the analysis
Decide on the inference model for each level
Do I want to generalize to a larger universe than just the units in the sample?
Different levels may have different inference models
Levels that exist in the design may be left out of the analysis model (but be sure you know what you are doing!)

6. Specifying Analyses Know the design Generally
Covariate effects should be fixed effects
Treatment effects should be random effects when the design permits it (e.g., randomized blocks designs)
Sometimes, there are too few units at a level to permit making it a random effect, even when this is theoretically sensible (e.g., 1 – 2 classrooms in a hierarchical design that assigns schools)

7. Specifying Analyses (Centering)
Centering is a transformation of the independent variables
In simple (simple random sample) designs, centering is just subtracting the mean
If Xi is the independent variable, the centered variable is
where is the mean of the Xi’s in the sample
Thus the mean of the centered variable is 0

8. Specifying Analyses (Centering)
What does centering do?
It changes the meaning of the intercept
Consider a regression equation
Yi = β0 + β1Xi + εi
And the regression equation with the centered predictor
Equating the centered and uncentered regression see that
Centering also changes the precision of estimate

9. Specifying Analyses (Centering)
In complex (multilevel) designs, there is more than one kind of centering:
Centering by grand mean
Centering by group mean
Grand mean centering is subtracting the grand mean
Group mean centering is subtracting the group mean
These centering methods affect interpretation of the intercept

10. Specifying Analyses (Grand Mean Centering)
What does grand mean centering do?
It changes the meaning of the intercept in the ith cluster
Consider a regression equation
Yij = β0i + β1iXij + εij
And the regression equation with the centered predictor
Equating the centered and uncentered regression see that
Centering also changes the precision of each estimate

11. Specifying Analyses (Group Mean Centering)
What does group mean centering do?
It changes the meaning of the intercept in the ith cluster
Consider a regression equation
Yij = β0i + β1iXij + εij
And the regression equation with the centered predictor
Equating the centered and uncentered regression see that
Centering also changes the precision of each estimate

12. Specifying Analyses (Group/Grand Mean Centering)
Generally it is useful to group mean center covariates
This permits the group mean to be used as a covariate at the group level
But the major reason to center is to increase precision of estimates, particularly estimates of variances of intercepts (which figure into standard errors of estimates of treatment effects)
Centering method should not greatly affect the results of analyses of experiments

13. Specifying Analyses (Weighting)
Weighting is sometimes used when samples are selected with the probability of selection in a stratum not proportionate to the size of the population
Sometimes people are tempted to weight in experiments so that the weighted proportions equal some preset (population) values
This is most defensible in experiments with randomized block designs
If treatment effects are heterogeneous, different weightings typically lead to different average treatment effects

14. Creating MDM Files First start the HLM Program by clicking on the icon
Next go to the file menu
Select make new MDM file,
Select from Stat Package input and select the relevant package
Then indicate that you want an HLM2 (two level analysis) or HLM3 (three level analysis)

15. Creating MDM Files Select Level 1 Variables
Select the radio button for nesting persons in groups
Under level 2 specification
Select “Browse” and select the file that has the level 2 variables
Select “Choose Variables” under level 2 specification
Select one variable as the ID variable (this variable is used to link the level 1 and level 2 data)
Select all of the other variables needed for the level 2 model
Select “Make MDM”
Then select “Check Stats”
Then select “Done”

16. Creating MDM Files Select Level 2 Variables
Select the radio button for nesting persons in groups
Select the file that has the level 1 variables
Select one variable as the ID variable (this variable is used to link the level 1 and level 2 data)
Select all of the other variables needed for the level 1 model
Indicate if there is missing data and if so, how it will be handled
When making the MDM file
When doing analyses