1. Conducting Meta-Analyses Marsha Sargeant, M.S. D esign A nd S tatistical A nalysis L aboratory University of Maryland, College Park Department of Psychology

2. 2 Overview of Presentation 1. What is a meta-analysis and why is it important? 2. Overview of procedures involved in conducting a quantitative meta-analysis 3. Database structure 4. Interpretation of effect sizes

3. 3 Meta-analysis Definition A statistical analysis of the summary findings of many empirical studies It’s quantitative! – Distinct from a meta-review

4. 4 Background Empirical findings grew exponentially in the middle 50 years of the 20th century – Multiplied beyond our ability to comprehend and integrate it – Hence a growing need to statistically and technically review, rather than through narrative

5. 5 Background Review of practices and methods of research reviewers and synthesizers in the social sciences (Jackson, 1978) Failure to report methods of reviewing

6. 6 Benefits of Meta-analyses Increased statistical power Identification of sources of variability across studies (e.g., inclusion of moderators) Detection of biases (e.g., Tower of Babel bias) Detection of deficiencies in design, analysis, or interpretation Ioannidis & Lau, 1999

7. 7 Limitations of Meta-analyses Cannot improve the original studies Method is frequently misapplied Can never follow the rules of science – Sources of bias are not controlled Ioannidis & Lau, 1999

8. 8 Rules of the Game It is quantitative There is no arbitrary exclusion of data File drawer effect – Dissertation research is research too! – Unpublished studies Meta-analysis seeks general conclusions – It is contradictory to think that we can only compare studies that are the same (if they were the same you wouldn’t need to compare them!) Glass, 2000

9. 9 Methodological Adequacy of Research Base Findings must be interpreted within the bounds of the methodological quality of the research base synthesized. Studies often cannot simply be grouped into “good” and “bad” studies. Some methodological weaknesses may bias the overall findings, others may merely add “noise” to the distribution. From “Practical Meta-analysis” by D.B. Wilson

10. 10 Confounding of Study Features Important study features are often confounding, obscuring the interpretive meaning of observed differences If the confounding is not severe and you have a sufficient number of studies, you can model “out” the influence of method features to clarify substantive differences From “Practical Meta-analysis” by D.B. Wilson

11. 11 Meta-analysis Overview Descriptives – Effect sizes (e.g., correlation coefficients) – Distribution and central tendency summarized Method section – Databases searched – Journals – What attempts were made to not have a biased search? – Criteria for inclusion – No effect studies Rosenthal, 2005

12. 12 Meta-analysis Overview Study quality – Use a weighting system – Use raters and non-dichotomous ratings to avoid weighter bias – Optimally raters should be blind to the results of the study – Ratings can be used as an adjustment on effect size or as a moderator to determine whether quality is related to obtained effect size Rosenthal, 2005

13. 13 Meta-analysis Overview Consider independence of studies – Treat non-independent studies as a single study with different dependent variables Recorded variables – Number, Age, Sex, Education, etc – Volunteer status – Laboratory or field study? – Randomized? – Method of data collection (e.g., interview vs questionnaire) – How constructs are operationalized – etc. Rosenthal, 2005

14. 14 Meta-analysis Overview Summarize recorded variables Study characteristics could all be potential moderators of outcome aside from those with particular meaning for the specific area of research Effect sizes (there are others) – R – Z r (Fisher’s r-Z transformation) – d family Cohen’s d Hedge’s g Glass’s delta Rosenthal, 2005

15. 15 Examples of Different Types of Effect Sizes Standardized mean difference – Group contrast research Treatment groups Naturally occurring groups – Inherently continuous construct Odds-ratio – Group contrast research Treatment groups Naturally occurring groups – Inherently dichotomous construct Correlation coefficient – Association between variables research From “Practical Meta-analysis - The Effect Size” by D.B. Wilson

16. 16 Interpreting Effect Size Results Cohen’s “Rules-of-Thumb” – standardized mean difference effect size small = 0.20 medium = 0.50 large = 0.80 – correlation coefficient small = 0.10 medium = 0.25 large = 0.40 – odds-ratio small = 1.50 medium = 2.50 large = 4.30 From “Practical Meta-analysis” by D.B. Wilson

17. 17 Interpreting Effect Size Results Rules-of-Thumb do not take into account the context of the intervention – a “small” effect may be highly meaningful for an intervention that requires few resources and imposes little on the participants – a small effect may be meaningful if the intervention is delivered to an entire population (prevention programs for school children) – small effects may be more meaningful for serious and fairly intractable problems From “Practical Meta-analysis” by D.B. Wilson

18. 18 Meta-analysis Overview Significance levels recorded – Recorded as the one-tailed standard normal deviates associated with p’s E.g., p’s of.10,.01.,.001 would be recorded as Z’s of 1.28, 2.33, and 3.09

19. 19 Meta-analysis Overview Report central tendency – Unwieghted mean effect size – Weighted mean effect size (weighting by size of study – can also use quality or other characteristic of interest) – Median – Proportion of studies showing effect sizes in the expected direction – Report number of studies reported on – Optional: total number of participants on which the weighted mean is based – Optional: median number of participants per obtained effect size

20. 20 Meta-analysis Overview Report variability – Standard deviation – Max and min effect size found at the 75th and 25th percentile – If normally distributed, the standard deviation is estimated at.75(Q3-Q1)

21. 21 Database Structure Database structures – The hierarchical nature of meta-analytic data – The familiar flat data file – The relational data file – Advantages and disadvantages of each – What about the meta-analysis bibliography? From “Practical Meta-analysis – Database Structure” by D.B. Wilson

22. 22 Database Structure Meta-analytic data is inherently hierarchical Any specific analysis can only include one effect size per study (or one effect size per sub-sample within a study) Analyses almost always are of a subset of coded effect sizes. Data structure needs to allow for the selection and creation of those subsets From “Practical Meta-analysis – Database Structure” by D.B. Wilson

23. 23 Example of a Flat Data File Note that there is only one record (row) per study Multiple ESs handled by having multiple variables, one for each potential ES. From “Practical Meta-analysis – Database Structure” by D.B. Wilson

24. 24 Database Structure Advantages and Disadvantages of a Single Flat File Structure Advantages – All data is stored in a single location – Familiar and easy to work with – No manipulation of data files prior to analysis Disadvantages – Only a limited number of ESs can be calculated per study – Any adjustments applied to ESs must be done repeatedly When to use – Interested in a small predetermined set of ESs – Number of coded variables is modest – Comfort level with a multiple data file structure is low From “Practical Meta-analysis – Database Structure” by D.B. Wilson

25. 25 Database Structure Example of Relational Data Structure (Multiple Related Flat Files) Note that a single record in the file above is “related” to five records in the file to the right Study Level Data File Effect Size Level Data File From “Practical Meta-analysis – Database Structure” by D.B. Wilson

26. 26 Database Structure Example of a More Complex Multiple File Data Structure Study Level Data FileOutcome Level Data File Effect Size Level Data File Note that study 100 has 3 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months) From “Practical Meta-analysis – Database Structure” by D.B. Wilson

27. 27 Database Structure Advantages & Disadvantages of Multiple Flat Files Data Structure Advantages – Can “grow” to any number of ESs – Reduces coding task (faster coding) – Simplifies data cleanup – Smaller data files to manipulate Disadvantages – Complex to implement – Data must be manipulated prior to analysis (creation of “working” analysis files) – Must be able to select a single ES per study for any given analysis When to use – Large number of ESs per study are possible From “Practical Meta-analysis – Database Structure” by D.B. Wilson

28. 28 What about Sub-Samples? So far I have assumed that the only ESs that have been coded were based on the full study sample What if you are interested in coding ESs separately for different sub-samples, such as, by gender or SES – Just say “no”! Often not enough of such data for meaningful analysis Complicates coding and data structure – Well, if you must, plan your data structure carefully Include a full sample effect size for each dependent measure of interest Place sub-sample in a separate data file From “Practical Meta-analysis – Database Structure” by D.B. Wilson

29. 29 Tips on Coding Paper Coding – include data file variable names on coding form – all data along left or right margin eases data entry Coding Directly into a Computer Database From “Practical Meta-analysis – Database Structure” by D.B. Wilson

30. 30 Example Screen from a Computerized Database for Direct Coding

31. 31 Coding Directly into a Computer Database Advantages – Avoids additional step of transferring data from paper to computer – Easy access to data for data cleanup – Data base can perform calculations during coding process (e.g., calculation of effect sizes) – Faster coding Disadvantages – Can be time consuming to set up the bigger the meta-analysis the bigger the payoff – Requires a higher level of computer skill From “Practical Meta-analysis – Database Structure” by D.B. Wilson

32. 32 Final Comments Meta-analysis – is a replicable and defensible method of synthesizing findings across studies – often points out gaps in the research literature, providing a solid foundation for the next generation of research on that topic – illustrates the importance of replication – facilitates generalization of the knowledge gain through individual evaluations From “Practical Meta-analysis” by D.B. Wilson

33. 33 Thank You! Email: msargeant@psyc.umd.edumsargeant@psyc.umd.edu Web: www.umd.academia.edu/MarshaSargeant