Meta-Analysis: Synthesizing evidence

Reminders Exam #3 (next class): Not cumulative, standard 40- question multiple choice Make-up LA and Paper rewrites (optional) Make-up Exam (optional): Short essays (60pts) Pick 2 of 4: Broad topics relevant to an entire unit (e.g., clinical research) or across units (e.g., validity) Short answer (100pts) Pick 6 of 12: Representative narrower topics (e.g., key considerations in designing RCTs)

Take-Home Messages of the Course Understand the value of research and experience a greater desire to engage in research in some capacity Understand p-values Understand the difference between a true “experiment” and a non-experiment Understand the importance of effect size (r, d)

Take-Home Messages of the Course Gain a greater tolerance for ambiguity Improve scientific writing and statistical skills Understand different methodologies as tools that can improve the quality of research Gain practical skills/accomplishments to foster academic and career goals Live a more science-driven life

Today’s Overview What does it mean to synthesize evidence? Why would we want to synthesize evidence? What is a meta-analysis?

Weighing and Synthesizing Evidence Any problem in life can be informed by scientific evidence Interpersonal dynamics, parenting, studying, worker satisfaction, psychotherapy, medical treatments, educational interventions, traffic safety, health A so-called science-driven life would use scientific evidence to make key life decisions Problem: Studies conflict (size/direction of effect,, confidence/p-value), and any particular study has strengths and weaknesses (the many reliabilities and validities)

Examples If you’re willing to shock your world view, read these articles: Djulbegovic, M., Beyth, R. J., Neuberger, M. M., Stoffs, T. L., Vieweg, J., Djulbegovic, B., & Dahm, P. (2010). Screening for prostate cancer: systematic review and meta-analysis of randomised controlled trials. BMJ, 341, c4543. Moyer, V. A. (2012). Screening for prostate cancer: US Preventive Services Task Force recommendation statement. Annals of Internal Medicine, 157(2), 120-134. Nelson, H. D., Fu, R., Cantor, A., Pappas, M., Daeges, M., & Humphrey, L. (2016). Effectiveness of breast cancer screening: systematic review and meta-analysis to update the 2009 US Preventive Services Task Force recommendation. Annals of Internal Medicine, 164(4), 244-255.

Weighing and Synthesizing Evidence Literature review Systematic literature review Meta-analysis Statistical approach for combining results from multiple studies Typically requires a systematic literature review to avoid bias Produces an average effect size (e.g., r, d, %) Bigger N  More power  More precision (e.g., better p-value, narrower confidence intervals)

Levels of Evidence (high to low) Meta-analysis of RCTs Experimental studies and RCTs Longitudinal observational studies and quasi-experimental studies Cross-sectional observational studies Single-subject designs Case series Case study Note. Other aspects of study quality could move a particular study up or down the list

Examples Mike’s current meta-analysis (separate file) Smith and Glass (1977). Meta-analysis of Psycho- therapy Outcomes.

Hauch et al. (2016). Does training improve the detection of deception?

Roberts et al. (2006). Patterns of mean-level change in personality traits across the life course.

Strengths and Limitations Excellent for synthesizing evidence, providing a starting point in a literature review, guiding policy Limitations Concern about publication bias and other biases in the systematic review process Concern about combining studies, samples, and interventions that are too heterogeneous