Week Seven

 The systematic and rigorous integration and synthesis of evidence is a cornerstone of EBP  Impossible to develop “best practice” guidelines, protocols, and procedures without organizing and evaluating research evidence through a systematic review

 Forms of systematic reviews: ◦ Narrative, qualitative integration (traditional review of quantitative or qualitative results) ◦ Meta-analysis (statistical integration of results) ◦ Metasynthesis (theoretical integration and interpretation of qualitative findings)

 Objectivity—statistical integration eliminates bias in drawing conclusions when results in different studies are at odds  Increased power—reduces risk of Type II error compared to single study  Increased precision—results in smaller confidence intervals than single studies

 Research question or hypothesis should be essentially identical across studies. ◦ The “fruit” problem—don’t combine apples and oranges!  Must be a sufficient knowledge base—must be enough studies of acceptable quality  Results can be varied but not totally at odds.

 Delineate research question or hypothesis to be tested.  Identify sampling criteria for studies to be included.  Develop and implement a search strategy.  Locate and screen sample of studies meeting the criteria.

 Appraise the quality of the study evidence.  Extract and record data from reports.  Formulate an analytic plan (i.e., make analytic decisions).  Analyze data according to plan.  Write a systematic review.

 Identify electronic databases to use.  Identify additional search strategies (e.g., ancestry approach).  Decide whether or not to pursue the gray literature (unpublished reports).  Identify keywords for the search: ◦ Think creatively and broadly.

 Meta-analysts must make decisions about handling study quality.  Approaches: ◦ Omit low-quality studies (e.g., in intervention studies, non-RCTs). ◦ Give more weight to high-quality studies. ◦ Analyze low- and high-quality studies to see if effects differ (sensitivity analyses).

 Evaluations of study quality can use: ◦ A scale approach (e.g., use a formal instrument to “score” overall quality) ◦ A component approach (code whether certain methodologic features were present or not, e.g., randomization, blinding, low attrition)

 Decisions include: ◦ What effect size index will be used? ◦ How will heterogeneity be assessed? ◦ Which analytic model will be used? ◦ Will there be subgroup (moderator) analyses? ◦ How will quality be addressed? ◦ Will publication bias be assessed?

 A central feature of meta-analysis is the calculation of an effect size index for each study that encapsulates the study results.  An ES index is computed for each study and then combined and averaged (often weighted for sample size).  Several different effect size (ES) indexes can be used.

 Major effect size indexes: ◦ d: the standardized difference between 2 groups (e.g., Es vs. Cs) on an outcome for which a mean can be calculated (e.g., BMI) ◦ Odds Ratio (OR): relative odds for two groups on a dichotomous outcome (e.g., smoke/not smoke) ◦ r: correlation between 2 continuous variables (e.g., age and depression)

 Results (effects) inevitably vary from one study to the next.  Major question: Is heterogeneity just random fluctuations? ◦ If “yes,” then a fixed effects model of analysis can be used. ◦ If “no,” then a random effects model should be used.  Heterogeneity can be formally tested but also can be assessed visually via a forest plot.

 Factors influencing variation in effects is usually explored via subgroup analysis (moderator analysis).  Do variations relate to: ◦ Participant characteristics (e.g., men vs. women)? ◦ Methods (e.g., RCTs vs. quasi-experiments)? ◦ Intervention characteristics (e.g., 3-week vs. 6- week intervention)?

 Nonpublished studies are more likely to have no effects or weak effects than published ones. ◦ So…excluding them could result in overestimating effects.  One approach: ◦ Compute a fail-safe number to see how many studies with 0 effect would be needed to change conclusions from significant to nonsignificant.

 One definition: The bringing together and breaking down of findings, examining them, discovering essential features, and combining phenomena into a transformed whole  Integrations that are more than the sum of the parts—novel interpretations of integrated findings

 Whether to exclude low-quality studies  Whether to integrate studies based in multiple qualitative traditions  Various typologies and approaches; differing terminology

 Similar to meta-analysis in many ways ◦ Formulate question ◦ Decide selection criteria, search strategy ◦ Search for and locate studies ◦ Extract data for analysis ◦ Formulate and implement an analysis approach ◦ Integrate, interpret, write up results

 Noblit and Hare (developed an approach for a meta-ethnography) ◦ Suggest a 7-phase approach ◦ Involves “translating” findings from qualitative studies into one another ◦ An “adequate translation maintains the central metaphors and/or concepts of each account” ◦ Final step is synthesizing the translations

 Paterson and colleagues’ approach involves three components: ◦ Meta-data analysis (analyzing and integrating the study findings) ◦ Meta-method (analyzing the methods and rigor of studies in the analysis) ◦ Meta-theory (analysis of the studies’ theoretical underpinnings)

 Sandelowski and Barrosa’s approach distinguishes studies that are summaries (no conceptual reframing) and syntheses (studies involving interpretation and metaphorical reframing).  Both summaries and syntheses can be used in a meta-summary, which can lay a foundation for a metasynthesis.

 Involve making an inventory of findings and can be aided by computing manifest effect sizes (effect sizes calculated from the manifest content in the studies in the review).  Two types: ◦ Frequency effect size ◦ Intensity effect size

 Frequency effect size ◦ Count the total number of findings across all studies in the review (specific themes or categories). ◦ Compute prevalence of each theme across all reports (e.g., the #1 theme was present in 75% of reports).  Intensity effect size ◦ For each report, compute how many of the total themes are included (e.g., report 1 had 60% of all themes identified).

 Can build on a meta-summary  But can only be done with studies that are syntheses (not summaries), because the purpose is to offer novel interpretations of interpretive findings—not just summaries of findings