MAKING SENSE OF DIAGNOSTIC META-ANALYSIS A VISUAL- GRAPHICAL FRAMEWORK Ben A. Dwamena, MD. Department of Radiology, University of Michigan Medical School/ Nuclear Medicine Service (115), VA Ann Arbor Health Care System, Ann Arbor. Francesca C. Dwamena, MD. Department of Medicine, Division of General Internal Medicine, Michigan State University College of Human Medicine, East Lansing.

EDUCATIONAL GOALS AND OBJECTIVES  PROVIDE A VISUAL/GRAPHIC FRAMEWORK FOR DIAGNOSTIC META- ANALYSIS THROUGH THE USE OF – Flow Charts – Contingency Tables – Forest Plots – Funnel Graphs – Scatter Diagrams – Summary Receiver Operator Characteristic Curves – Etc

DIAGNOSTIC META- ANALYSES  Improve quality of future primary studies by identifying methodological deficiencies  Identify reasons for variation in reported results  Generate valid summary estimates of diagnostic performance

VISUAL DISPLAYS/GRAPHS  Provide more user-friendly summaries of large quantitative data sets  Preliminary data exploration before definite data synthesis  Clarify difficult statistical concepts and interpretation

GENERAL ARCHITECTURE  Formulate Question  Develop Search Strategy and Retrieve Articles  Select Eligible Studies and Assess Quality  Extract Data and Calculate Individual Summary Measures  Choose Model for Pooling  Investigate Heterogeneity and Biases

DATA SOURCES VISUAL/GRAPHIC DISPLAYS BASED ON  Dwamena BA, Sonnad SS, et al. Metastases From Non-small Cell Lung Cancer: Mediastinal Staging In The 1990s- A Meta-analytic Comparison Of PET and CT.Radiology 1999; 213:  Published Work of Other Investigators in the Field Based on Either Original or Simulated Data

RESEARCH QUESTION  GENERAL EXAMPLE: How Accurate Is a Sign, Symptom, or Diagnostic Test in Predicting the True Diagnostic Category of a Patient?  RELEVANT QUESTION: Addresses Population or Patient Group, Diagnostic Intervention, Disease of Interest

FINDING RELEVANT STUDIES  SEARCH FOR EXISTING REVIEWS  FIND PUBLISHED PRIMARY STUDIES Break down research question into components Use appropriate synonyms Use electronic databases, hand searching,etc  LOOK FOR UNPUBLISHED PRIMARY STUDIES ( write to experts, search registries for completed/ongoing trials)

BREAKING QUESTION DOWN INTO COMPONENTS “What is the accuracy of fecal occult blood test for detection of colorectal cancer? “ may be represented by a VENN DIAGRAM:

RECOMMENDED SEARCH STRATEGY REGARDING TEST PERFORMANCE Deville WL et. al. BMC Medical Research Methodology 2002, 2:9-22

QUALITY CITERIA  PATIENT SELECTION: Consecutive vs. Non-consecutive or convenience sample  SPECTRUM: Clinically relevant population versus case-control  REFERENCE STANDARD: Full vs. Partial reporting of cut-off value  DIAGNOSTIC TEST: Full vs. partial reporting of cut-off value  DATA COLLECTION: Prospective versus Retrospective versus unknown  DETAILS OF POPULATION: Sufficient versus Insufficient  VERIFICATION: Complete versus different reference tests versus incomplete  INTERPRETATION OF RESULTS: Blinded versus Unblinded

METHODOLOGICAL STANDARDS  Quality of Each Selected Paper Should Be Assessed Independently by at Least Two Reviewers.  Chance-adjusted Agreement Should Be Reported and Disagreements Solved by Consensus or Arbitration.  To Improve Agreement, Reviewers Should Pilot Their Quality Assessment Tools in a Subset of Included Studies or Studies Evaluating a Different Diagnostic Test

METHODOLOGICAL STANDARDS

FLOW CHART OF STUDY RETRIEVAL AND SELECTION

DATA EXTRACTION  Info About the Participants Included in the Study, Time of Data Collection and the Testing Procedures.  The Cut-off Point Used in Dichotomous Testing Reasons and the Number of Participants Excluded Because of Indeterminate Results or Unfeasibility.  Extracted Information May Be Used in Subgroup Analyses and Statistical Pooling.

DATA EXTRACTION  Multiple Reviewers Should Independently Extract the Required Information.  Obtain Data Construct the Diagnostic 2 × 2 Table: Absolute Numbers in the Four Cells Are Needed.  Obtain Totals 'Diseased' and 'Non-diseased' to Calculate Prior Probability (Pre-test Probability) From Recalculated Sensitivity, Specificity, Likelihood Ratios, Predictive Values

CONTINGENCY TABLE FOR EXTRACTION OF TEST DATA

DIAGNOSTIC VS. TREATMENT TRIAL  True Positives =Experimental Group With the Monitored Outcome Present (a).  False Positives = Control Group With the Outcome Present (b).  False Negatives=experimental Group With the Outcome Absent (c).  True Negatives Are the Control Group With the Outcome Absent (d).

DIAGNOSTIC VS. TREATMENT TRIAL  Relative risk in experimental group {[a/(a + c)]/[b/(b+ d)]} =Likelihood Ratio for a Positive Test.  Relative Risk in Control Group = Likelihood Ratio for a Negative Test.  The Expression for the Odds Ratio (OR) Is (a x d)/(b x c).

CONTINGENCY DATA FOR NSCLC PET STUDY

CHOICE OF MODEL AND INDEX FOR POOLING OF TEST PERFORMANCE

SEARCHING FOR THRESHOLD EFFECT  Test for the Presence of Threshold Effect Between Studies by Calculating a Spearman Correlation Coefficient Between Sensitivity and Specificity of All Included Studies  A Spearman Correlation of < -0.6, Suggests Evidence of Interdependence of Sensitivity and Specificity, and SROC Curves Should Be Constructed or ROC Curves Can Be Pooled

SEARCHING FOR HOMOGENEITY  Perform Chi-square or Fisher’s Test for Small Number Studies.  If Sensitivity and Specificity Are Homogeneous, and Show No Threshold Effect, They Can Be Pooled by Fixed Effect Model.  If Heterogeneity Is Present, Restrict the Analysis to a Qualitative Overview; Pool Data From Homogeneous Sub-groups; Use Random Effect Model.

SROC PROCEDURE 1 SCATTER PLOT TPR VS. FPR Visualization of range of results from primary studies 2 REGRESSION OF D ON S Straight lines fitted to estimate (a) best fit to the data (b) remove effect of possible relationship between results and positivity threshold 3 BACKTRANSFORMATI ON OF REGRESSION TO CONVENTIONAL AXES Presentation of combined results into a single ROC curve

LINEAR REGRESSION ANALYSIS  Logit transformations of the TP rate (sensitivity) and FP rate (1 - specificity). D=ln(DOR) =logit(TPR) – logit(FPR)  Differences in logit transformations, D, regressed on sums of logit transformations, S. S=logit(TPR)+logit(FPR)  Logit(TPR)=natural log odds of a TP result and logit(FPR) =natural log of the odds of a FP test result.

LINEAR REGRESSION MODELS  ORDINARY LEAST SQUARES METHOD Studies are weighted equally  WEIGHTED LEAST SQUARES METHOD Weighted by the inverse variance weights of the diagnostic odds ratio, or simply the sample size  ROBUST-RESISTANT METHOD Minimizes the influence of outliers

LINEAR REGRESSION PLOT

SUMMARY ROC CURVE  Back transformation of logistic regression to conventional axes of sensitivity [TPR] vs. (1 – specificity) [FPR]) with the equation  TPR = 1/{1 + exp[- a/(1 - b )]} [(1 - FPR)/(FPR)] (1 + b )/(1 - b ).  Slope (b) and intercept (a) are obtained from the linear regression analyses

SUMMARY ROC CURVE

FOREST PLOT OF STUDY-SPECIFIC AND SUMMARY SENSITIVITY AND SPECIFICITY

FIXED EFFECTS META- ANALYSIS  Assumes Same Diagnostic Accuracy in All Studies  Variation in Sensitivity and Specificity From Published Reports Due to Random Error/chance Threshold Variation

FIXED EFFECTS META- ANALYSIS

RANDOM EFFECTS META- ANALYSIS  Assumes Diagnostic Accuracy Varies From Study to Study  Variation in of Reported Accuracy Estimates Are Randomly Distributed About Some Central Value Represented by SROC.  Variation Due to Stage of Disease, Clinical Presentation, Prevalence of Disease, Study Design Etc.

RANDOM EFFECTS META- ANALYSIS

WEIGHTED HISTOGRAM BREAST CANCER DATA

DEALING WITH HETEROGENEITY  Repeat Analysis Analysis After Excluding Outliers  Conduct Analysis With Predefined Subgroups.  Use Analysis of Variance With the Lndor As Dependent Variable and Categorical Variables for Subgroups As Factors to Look for Differences Among Subgroups;  Construct Multivariate Models to Search for the Independent Effect of Study Characteristics

GALBRAITH PLOT  Standardized Log-odds Ratio Plotted Against the Reciprocal of the Standard Error.  Small Studies/less Precise Results Appear on the Left Side and the Largest Trials on the Right End.  A Regression Line, Through the Origin, Represents the Overall Log-odds Ratio.  Lines +/- 2 Above Regression Line, Represent the 95 Per Cent Boundaries of the Overall Log-odds Ratio.  The Majority of Results Should Lie in This Area in the Absence of Heterogeneity.

GALBRAITH PLOT

FUNNEL DIAGRAM  A Funnel Diagram (A.K.A. Funnel Plot, Funnel Graph) Is a Special Type of Scatter Plot With an Estimate Sample Size on One Axis and Effect-size Estimate on the Other Axis.  Based on the Well Known Statistical Principle That Sampling Error Decreases As Sample Size Increases.  Used to Search for Publication Bias and to Test Whether All Studies Come From a Single Population.

SIMULATED FUNNEL PLOTS FOR EXPLORING PUBLICATION BIAS

FUNNEL PLOT REGRESSION

EGGER’S REGRESSION METHOD FOR DETECTING PUBLICATION BIAS

FUNNEL PLOT OF NSCLC PET DATA

NORMAL QUANTILE PLOTS

NORMAL QUANTILE PLOT OF NSCLC PET DATA

NORMAL QUANTILE PLOT OF AXILLARY BREAST CANCER PET DATA

ODDS RATIO SUBGROUP ANALYSIS

SROC SUBGROUP ANALYSIS

SOFTWARE  METATEST (Dr Lau, NEMC, Boston) SROC Curve Analysis  METAWIN (Sinauer Associates, Sunderland, MA) Scatter, Funnel, Normal Quantile, Forest, Weight Histogram, Radial (Galbraith) and Cummulative Meta-analysis Plots  STATSDIRECT (StatsDirect Ltd, Herts, UK) Forest, Funnel and L’abbe Plots