Download presentation
Presentation is loading. Please wait.
2
MAKING SENSE OF DIAGNOSTIC META-ANALYSIS A VISUAL- GRAPHICAL FRAMEWORK Ben A. Dwamena, MD. Department of Radiology, University of Michigan Medical School/ Nuclear Medicine Service (115), VA Ann Arbor Health Care System, Ann Arbor. Francesca C. Dwamena, MD. Department of Medicine, Division of General Internal Medicine, Michigan State University College of Human Medicine, East Lansing.
3
EDUCATIONAL GOALS AND OBJECTIVES PROVIDE A VISUAL/GRAPHIC FRAMEWORK FOR DIAGNOSTIC META- ANALYSIS THROUGH THE USE OF – Flow Charts – Contingency Tables – Forest Plots – Funnel Graphs – Scatter Diagrams – Summary Receiver Operator Characteristic Curves – Etc
4
DIAGNOSTIC META- ANALYSES Improve quality of future primary studies by identifying methodological deficiencies Identify reasons for variation in reported results Generate valid summary estimates of diagnostic performance
5
VISUAL DISPLAYS/GRAPHS Provide more user-friendly summaries of large quantitative data sets Preliminary data exploration before definite data synthesis Clarify difficult statistical concepts and interpretation
6
GENERAL ARCHITECTURE Formulate Question Develop Search Strategy and Retrieve Articles Select Eligible Studies and Assess Quality Extract Data and Calculate Individual Summary Measures Choose Model for Pooling Investigate Heterogeneity and Biases
7
DATA SOURCES VISUAL/GRAPHIC DISPLAYS BASED ON Dwamena BA, Sonnad SS, et al. Metastases From Non-small Cell Lung Cancer: Mediastinal Staging In The 1990s- A Meta-analytic Comparison Of PET and CT.Radiology 1999; 213:530- 36. Published Work of Other Investigators in the Field Based on Either Original or Simulated Data
8
RESEARCH QUESTION GENERAL EXAMPLE: How Accurate Is a Sign, Symptom, or Diagnostic Test in Predicting the True Diagnostic Category of a Patient? RELEVANT QUESTION: Addresses Population or Patient Group, Diagnostic Intervention, Disease of Interest
9
FINDING RELEVANT STUDIES SEARCH FOR EXISTING REVIEWS FIND PUBLISHED PRIMARY STUDIES Break down research question into components Use appropriate synonyms Use electronic databases, hand searching,etc LOOK FOR UNPUBLISHED PRIMARY STUDIES ( write to experts, search registries for completed/ongoing trials)
10
BREAKING QUESTION DOWN INTO COMPONENTS “What is the accuracy of fecal occult blood test for detection of colorectal cancer? “ may be represented by a VENN DIAGRAM:
11
RECOMMENDED SEARCH STRATEGY REGARDING TEST PERFORMANCE Deville WL et. al. BMC Medical Research Methodology 2002, 2:9-22
12
QUALITY CITERIA PATIENT SELECTION: Consecutive vs. Non-consecutive or convenience sample SPECTRUM: Clinically relevant population versus case-control REFERENCE STANDARD: Full vs. Partial reporting of cut-off value DIAGNOSTIC TEST: Full vs. partial reporting of cut-off value DATA COLLECTION: Prospective versus Retrospective versus unknown DETAILS OF POPULATION: Sufficient versus Insufficient VERIFICATION: Complete versus different reference tests versus incomplete INTERPRETATION OF RESULTS: Blinded versus Unblinded
13
METHODOLOGICAL STANDARDS Quality of Each Selected Paper Should Be Assessed Independently by at Least Two Reviewers. Chance-adjusted Agreement Should Be Reported and Disagreements Solved by Consensus or Arbitration. To Improve Agreement, Reviewers Should Pilot Their Quality Assessment Tools in a Subset of Included Studies or Studies Evaluating a Different Diagnostic Test
14
METHODOLOGICAL STANDARDS
15
FLOW CHART OF STUDY RETRIEVAL AND SELECTION
16
DATA EXTRACTION Info About the Participants Included in the Study, Time of Data Collection and the Testing Procedures. The Cut-off Point Used in Dichotomous Testing Reasons and the Number of Participants Excluded Because of Indeterminate Results or Unfeasibility. Extracted Information May Be Used in Subgroup Analyses and Statistical Pooling.
17
DATA EXTRACTION Multiple Reviewers Should Independently Extract the Required Information. Obtain Data Construct the Diagnostic 2 × 2 Table: Absolute Numbers in the Four Cells Are Needed. Obtain Totals 'Diseased' and 'Non-diseased' to Calculate Prior Probability (Pre-test Probability) From Recalculated Sensitivity, Specificity, Likelihood Ratios, Predictive Values
18
CONTINGENCY TABLE FOR EXTRACTION OF TEST DATA
19
DIAGNOSTIC VS. TREATMENT TRIAL True Positives =Experimental Group With the Monitored Outcome Present (a). False Positives = Control Group With the Outcome Present (b). False Negatives=experimental Group With the Outcome Absent (c). True Negatives Are the Control Group With the Outcome Absent (d).
20
DIAGNOSTIC VS. TREATMENT TRIAL Relative risk in experimental group {[a/(a + c)]/[b/(b+ d)]} =Likelihood Ratio for a Positive Test. Relative Risk in Control Group = Likelihood Ratio for a Negative Test. The Expression for the Odds Ratio (OR) Is (a x d)/(b x c).
21
CONTINGENCY DATA FOR NSCLC PET STUDY
22
CHOICE OF MODEL AND INDEX FOR POOLING OF TEST PERFORMANCE
23
SEARCHING FOR THRESHOLD EFFECT Test for the Presence of Threshold Effect Between Studies by Calculating a Spearman Correlation Coefficient Between Sensitivity and Specificity of All Included Studies A Spearman Correlation of < -0.6, Suggests Evidence of Interdependence of Sensitivity and Specificity, and SROC Curves Should Be Constructed or ROC Curves Can Be Pooled
24
SEARCHING FOR HOMOGENEITY Perform Chi-square or Fisher’s Test for Small Number Studies. If Sensitivity and Specificity Are Homogeneous, and Show No Threshold Effect, They Can Be Pooled by Fixed Effect Model. If Heterogeneity Is Present, Restrict the Analysis to a Qualitative Overview; Pool Data From Homogeneous Sub-groups; Use Random Effect Model.
25
SROC PROCEDURE 1 SCATTER PLOT TPR VS. FPR Visualization of range of results from primary studies 2 REGRESSION OF D ON S Straight lines fitted to estimate (a) best fit to the data (b) remove effect of possible relationship between results and positivity threshold 3 BACKTRANSFORMATI ON OF REGRESSION TO CONVENTIONAL AXES Presentation of combined results into a single ROC curve
26
LINEAR REGRESSION ANALYSIS Logit transformations of the TP rate (sensitivity) and FP rate (1 - specificity). D=ln(DOR) =logit(TPR) – logit(FPR) Differences in logit transformations, D, regressed on sums of logit transformations, S. S=logit(TPR)+logit(FPR) Logit(TPR)=natural log odds of a TP result and logit(FPR) =natural log of the odds of a FP test result.
27
LINEAR REGRESSION MODELS ORDINARY LEAST SQUARES METHOD Studies are weighted equally WEIGHTED LEAST SQUARES METHOD Weighted by the inverse variance weights of the diagnostic odds ratio, or simply the sample size ROBUST-RESISTANT METHOD Minimizes the influence of outliers
28
LINEAR REGRESSION PLOT
29
SUMMARY ROC CURVE Back transformation of logistic regression to conventional axes of sensitivity [TPR] vs. (1 – specificity) [FPR]) with the equation TPR = 1/{1 + exp[- a/(1 - b )]} [(1 - FPR)/(FPR)] (1 + b )/(1 - b ). Slope (b) and intercept (a) are obtained from the linear regression analyses
30
SUMMARY ROC CURVE
31
FOREST PLOT OF STUDY-SPECIFIC AND SUMMARY SENSITIVITY AND SPECIFICITY
32
FIXED EFFECTS META- ANALYSIS Assumes Same Diagnostic Accuracy in All Studies Variation in Sensitivity and Specificity From Published Reports Due to Random Error/chance Threshold Variation
33
FIXED EFFECTS META- ANALYSIS
34
RANDOM EFFECTS META- ANALYSIS Assumes Diagnostic Accuracy Varies From Study to Study Variation in of Reported Accuracy Estimates Are Randomly Distributed About Some Central Value Represented by SROC. Variation Due to Stage of Disease, Clinical Presentation, Prevalence of Disease, Study Design Etc.
35
RANDOM EFFECTS META- ANALYSIS
36
WEIGHTED HISTOGRAM BREAST CANCER DATA
37
DEALING WITH HETEROGENEITY Repeat Analysis Analysis After Excluding Outliers Conduct Analysis With Predefined Subgroups. Use Analysis of Variance With the Lndor As Dependent Variable and Categorical Variables for Subgroups As Factors to Look for Differences Among Subgroups; Construct Multivariate Models to Search for the Independent Effect of Study Characteristics
38
GALBRAITH PLOT Standardized Log-odds Ratio Plotted Against the Reciprocal of the Standard Error. Small Studies/less Precise Results Appear on the Left Side and the Largest Trials on the Right End. A Regression Line, Through the Origin, Represents the Overall Log-odds Ratio. Lines +/- 2 Above Regression Line, Represent the 95 Per Cent Boundaries of the Overall Log-odds Ratio. The Majority of Results Should Lie in This Area in the Absence of Heterogeneity.
39
GALBRAITH PLOT
40
FUNNEL DIAGRAM A Funnel Diagram (A.K.A. Funnel Plot, Funnel Graph) Is a Special Type of Scatter Plot With an Estimate Sample Size on One Axis and Effect-size Estimate on the Other Axis. Based on the Well Known Statistical Principle That Sampling Error Decreases As Sample Size Increases. Used to Search for Publication Bias and to Test Whether All Studies Come From a Single Population.
41
SIMULATED FUNNEL PLOTS FOR EXPLORING PUBLICATION BIAS
42
FUNNEL PLOT REGRESSION
43
EGGER’S REGRESSION METHOD FOR DETECTING PUBLICATION BIAS
44
FUNNEL PLOT OF NSCLC PET DATA
45
NORMAL QUANTILE PLOTS
47
NORMAL QUANTILE PLOT OF NSCLC PET DATA
48
NORMAL QUANTILE PLOT OF AXILLARY BREAST CANCER PET DATA
49
ODDS RATIO SUBGROUP ANALYSIS
50
SROC SUBGROUP ANALYSIS
51
SOFTWARE METATEST (Dr Lau, NEMC, Boston) SROC Curve Analysis METAWIN (Sinauer Associates, Sunderland, MA) Scatter, Funnel, Normal Quantile, Forest, Weight Histogram, Radial (Galbraith) and Cummulative Meta-analysis Plots STATSDIRECT (StatsDirect Ltd, Herts, UK) Forest, Funnel and L’abbe Plots
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.