Presentation is loading. Please wait.

Presentation is loading. Please wait.

Presented by Nancy Vargas.

Similar presentations


Presentation on theme: "Presented by Nancy Vargas."— Presentation transcript:

1 Presented by Nancy Vargas

2 Content Publication Bias (Chapter 30) Outcome Selection Bias
Cochrane Collaboration's Tool for Assessing Risk of Bias

3 Publication Bias What’s the problem?
Published studies are unrepresentative of all completed studies What’s the phenomenon? Published studies are often more statistically significant than unpublished studies

4 Publication Bias How does it affect MA’s?
We are more likely to include published studies in our MA which depends on the literature What are the arguements against grey literature? They are lower quality studies Published studies go through quality control 1)Leads to an upward bias in the summary effect 3) generally indefensible to conduct a synthesis that categorically excludes unpublished research reports such as conference papers, thesis, technical reports Are any

5 Publication Bias Discussion Questions
Should publication status determine quality? Why or why not? Are any of you excluding grey literture? Why or why not? 1)Leads to an upward bias in the summary effect 3) generally indefensible to conduct a synthesis that categorically excludes unpublished research reports such as conference papers, thesis, technical reports

6 Publication Bias How can we determine if there is publication bias when we run our analysis? In report, always include assessment of publication bias. Trim and fill-A problem with this method is that it depends strongly on the assumptions of the model for why studies are missing, and the algorithm for detecting asymmetry can be influenced by one or two aberrant studies.

7 Case Study: Physical Activity and Psychological Well-Being in Advanced Age: A Meta-Analysis of Intervention Studies Intercept=1.35 SE=.310 z=4.35 p<.001 *cited 657 times There is significant asymmetry Is there a possibility of publication bias? Yes, because the lower left corner is missing studies meaning you are not including negative effects or null studies What would be our next steps? Fail safe N to determine how many unpublished studies showing a null result are required to change the MA from significant to non significant Trim and Fill Method (What is our best estimate of the unbiased effect size? )- removes the most extreme small studies from the positive side of the funnel plot, re-computing the effect size at each iteration until the funnel plot is symmetric about the (new) effect size. In theory, this will yield an unbiased estimate of the effect size.

8 Outcome Selection Bias
Describe what is happening in the picture. How does this relate to outcome selection bias? 0.05 0.75 0.01 0.04 0.12

9 Outcome Selection Bias
What is it? Selective reporting of outcomes within published studies Why does this problem exist? Publication bias! Can you name some examples? AKA Within-study selective reporting bias, selection, on the basis of the results, of a subset of the analyses undertaken to be included in a study publication. lack of statistical significance, journal space restrictions and lack of clinical importance.

10 Outcome Selection Bias
How to Identify Outcome Selection Bias Direct statements made by authors Indirectly if certain outcome measure was suggested but not reported and previous knowledge in certain tool/scales 1)Only most sig. results reported

11 Outcome Selection Bias
For Individual Studies Results may also nonreported due to lack of statistical significance Lack of complete subgroup data For MA Use funnel plot to detect whether there are studies eligible for the meta-analysis that do not provide data to be included in the meta-analysis (Trim and Fill) Impact on MA is undetermined, more research is needed 1)Only most sig. results reported 2)Same as above but with subgroup data, we reduce bias in MA with moderator variables (population characteristics?)

12 Outcome Selection Bias
Solutions Sensitivity analyses Acknowledge your bias Development of standard outcomes for certain conditions Publication of raw data Report changes in study protocol 1) allows the impact of possible selection of measures or subgroups for publication to be assessed 2)If reviewer is influenced by their prior knowledge of what has been reported or what has been found to be interesting in the studies that they will meta-analyze in regards to outcomes. 5)Change in primary outcome

13 Criteria for Judging Risk of Bias
Cochrane Collaboration's tool for assessing risk of bias Two part tool discussing seven different domains sequence generation allocation concealment blinding of participants and personnel blinding of outcome assessment incomplete outcome data selective outcome reporting other issues These domains are included in a risk of bias table the first part of the tool describes what was reported to have happened in the study, in sufficient detail to support a judgement about the risk of bias. The second part of the tool assigns a judgement relating to the risk of bias for that entry.  Purple:  each be addressed in the tool by a single entry for each study Black: two or more entries may be used because assessments generally need to be made separately for different outcomes (or for the same outcome at different time points). (?) Other: can be assessed as a single entry for studies as a whole

14 Criteria for Judging Risk of Bias
Reports are developed in RevMan 5 Free to use and can be downloaded here (no support for non-members?) Need to learn how to input studies. Once complete, risk of bias tables, graphs, and summaries can be developed Rev man stands for review manager

15 Criteria for Judging Risk of Bias
judgements should be categorized as ‘Low risk’ of bias, ‘High risk’ of bias or ‘Unclear risk’ of bias consider the risk of material bias, bias of sufficient magnitude to have a notable impact on the results The support for judgement provides a succinct summary from which judgements of risk of bias can be made, and aims to ensure transparency in how these judgements are reached. the support for judgement should include verbatim quotes from reports or correspondence it may include a summary of known facts, or a comment from the review authors. include other information that influences any judgements made (such as knowledge of other studies performed by the same investigators) Ambiguous quotes: Probably done or not done and provide rationale

16 Criteria for Judging Risk of Bias
Two figures may be generated using RevMan for inclusion in a published review. First, a ‘Risk of bias graph’ figure illustrates the proportion of studies with each of the judgements (‘Low risk’, ‘High risk’, ‘Unclear risk’ of bias) for each entry in the tool 

17 Criteria for Judging Risk of Bias
Second, a ‘Risk of bias summary’ figure presents all of the judgements in a cross-tabulation of study by entry Therefore, the major approach to incorporating risk of bias assessments in Cochrane reviews is to restrict meta-analyses to studies at low (or lower) risk of bias, or to stratify studies according to risk of bias.

18 Criteria for Judging Risk of Bias
How do we incorporate this into our results section? Risk of bias in included studies section of our results general risk of bias in results of the included studies variability across studies important flaws in individual studies. Include Risk of bias table and a concise summary *Assessment criteria for studies would be in methods   

19 Criteria for Judging Risk of Bias
Summary of risk of bias assessment for the primary outcomes under the following headings is needed for large reviews. Allocation Blinding Incomplete outcome data Selective reporting Other potential sources of bias    For large reviews, aspects of the assessment of risk of bias may be summarized for the primary outcomes under the following headings. Allocation [recommended, level 3 heading] A summary of how allocation sequences were generated and attempts to conceal allocation of intervention assignment should be summarized briefly here, along with any judgements concerning the risk of bias that may arise from the methods used. Blinding A brief summary of who was blinded or masked during the conduct and analysis of the studies should be reported here. Implications of blinding of outcome assessment may be different for different outcomes, so these may need to be addressed separately. Judgements concerning the risk of bias associated with blinding should be summarized. Incomplete outcome data The completeness of data should be summarized briefly here for each of the main outcomes. Concerns of the review authors over exclusion of participants and excessive (or differential) drop-out should be reported. Selective reporting Concerns over the selective availability of data may be summarized briefly here, including evidence of selective reporting of outcomes, time-points, subgroups or analyses. Other potential sources of bias Any other potential concerns should be summarized her

20 Criteria for Judging Risk of Bias
Allocation How allocation sequences were generated Attempts to conceal allocation of intervention assignment Judgements concerning the risk of bias that may arise from the methods Blinding Who was blinded or masked during the conduct and analysis of the studies Implications of blinding for each outcome Judgements concerning the risk of bias    Allocation [recommended, level 3 heading] A summary of how allocation sequences were generated and attempts to conceal allocation of intervention assignment should be summarized briefly here, along with any judgements concerning the risk of bias that may arise from the methods used. Blinding A brief summary of who was blinded or masked during the conduct and analysis of the studies should be reported here. Implications of blinding of outcome assessment may be different for different outcomes, so these may need to be addressed separately. Judgements concerning the risk of bias associated with blinding should be summarized.

21 Criteria for Judging Risk of Bias
Incomplete Outcome Data For main outcomes, summarize completeness of data Concerns over exclusion/high drop out rates Selective Reporting Summary of concerns over the selective availability of data (outcomes, time-points, subgroups or analyses) Other potential sources of bias    Incomplete outcome data [recommended, level 3 heading] The completeness of data should be summarized briefly here for each of the main outcomes. Concerns of the review authors over exclusion of participants and excessive (or differential) drop-out should be reported. Selective reporting Concerns over the selective availability of data may be summarized briefly here, including evidence of selective reporting of outcomes, time-points, subgroups or analyses. Other potential sources of bias Any other potential concerns should be summarized her

22 GRADE Judgement on quality of evidence for each outcome
   GRADE reflects how confident we are that an estimate is close to the truth

23 GRADE    Review of GRADE Study limitations (risk of bias) 2. Indirectness of evidence 3. Inconsistency of results 4. Imprecision 5. Publication bias No concerns – Serious −1 level – Very serious −2 levels

24


Download ppt "Presented by Nancy Vargas."

Similar presentations


Ads by Google