Principles of Research Synthesis Benjamin Djulbegovic, M.D.,PhD. H. Lee Moffitt Cancer Center University of South Florida San Francisco Radiation Oncology Conference February 28 to March 2, 2003

I The need for research synthesis

The need for research synthesis Health care decision makers need to access research evidence to make informed decisions on diagnosis, treatment and health care management for both individual patients and populations. There are few important questions in health care which can be informed by consulting the result of a single empirical study.

II The problems with traditional review articles

The need for research synthesis Importance of review articles –Review articles in medical journals summarize large amounts of information on a particular topic and therefore are useful and popular source of information for health care professionals – review articles have the highest impact factor which means, that research, practice and policy- decisions are significantly influenced by review articles

Science of research synthesis: problems with traditional review articles Personal views on the available body of evidence Selection bias and selective citations monster –Has been pervasive in medicine, economics and social sciences Can obscure up to 40-60% of true intervention’s effect In 2000, Nobel prize in Economic Science was awarded to James Heckman of the University of Chicago for his analysis of selection bias, which in turn profoundly affected applied research in economics as well as in other social sciences Lack of reproducibility –that is, the lack of the key scientific criterion

Selective citation bias: blind men and elephant

Critique of reviews of chemotherapy for ovarian cancer Crx superior (by Qualitative analysis) 48/53 Search strategy3/53 Inclusion/exclusion criteria2/53 Validity assessment1/53 Quantitative Assessment3/53 Courtesy of Dr. C. Williams

Quality of Review Article:158 articles only 2 met all 10 criteria. Ann Intern Med 1999;131:

Research Synthesis: terminology Systematic review. The application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic. Meta- analysis may be, but is not necessary, used as part of this process. Meta-analysis. The statistical synthesis of the data from separate but similar, i.e. comparable studies, leading to a quantitativ summary of the pooled results.

Key Distinctions Between Narrative and Systematic Reviews, by Core Features of Such Reviews Core FeatureNarrative Review Systematic Review Study question Data sources and search strategy Selection of articles for study Article review or appraisal Study quality Synthesis Inferences Often broad in scope Which databases were searched and search strategy are not typically provided. Not usually specified, potentially biased. Variable, depending onwho is conducting the review. If assessed, may not use formal quality assessment. Often a qualitative summary. Sometimes evidence-based. Often a focused clinical question. Comperehensive search of many databases as well as so-called gray literature. Explicit search strategy Criterion-basedSelection, uniformlyapplied. Rigorous critical appraisal, typically usinga data extraction form. Some assessment of quality is almost always included as part of the data extraction process. Quantitative summary (meta-analysis) if the data can be appropriately pooled ; qualitative otherwise. Usually evidence-based

Principles of reliable detection of the effects of health care interventions Methods to reduce bias Methods to reduce statistical imprecision

III Principles of systematic reviews and meta-analysis

Principle #1:the need to consider the totality of evidence –“The world can be only considered as the totality of facts…for the totality of facts determines what is the case, and also whatever is not the case” L. Wittgenstein (“Tractus logico-philosophicus”), 1921

Principle #2: requirement for reproducibility Transparent, explicit and systematic approach in identifying and synthesizing evidence –Methods for search for evidence –Inclusion and exclusion criteria –Quality assessment

Steps of a Systematic Review Search of personal files Systematic manual searches of key journals Computerized Databases Review of reference lists of articles Consultation with experts Identify studies Review for relevance Evaluate methodological quality Extract data Analyze data Reject Draw Conclusions RelevantNot Relevant

the QUOROM statement

Principles of reliable detection of the effects of health care interventions Systematic bias must be < the effect of intervention which we are trying to detect –The need for the totality of evidence (published and unpublished) Random errors (play of chance) must be < the effect of intervention which we are trying to detect –uncertainty/imprecision reduced by pooling all available data –Need for large number of patients/events

Rationale for (quantitative) synthesis of all available evidence The rationale for pooling data is clinical and not statistical Similar interventions for similar conditions will produce the similar effects (i.e. in the same direction) –While the effect size may not be the same, it will rarely be in the opposite directions –Meta-analysis attempts to show direction of the effect (i.e. help establish generalisability of the effect)

Disease population RCT1 RCT2RCT3 0.8 Test for heterogeneity chi square; df 2, p=0.1 Test for overall effect Z; p= Favors new treatmentFavors control RR (95% CI Fixed) 10 Relative risk A)

Disease population RCT1 RCT2RCT3 0.8 Test for heterogeneity chi square; df 2, p=0.02 Test for overall effect Z; p= Favors new treatment Favors control RR (95% CI Fixed) 10 Relative risk B)

Calculate “observed minus expected” for each trial TreatedControl Dead Alive Obs=15 Obs=10 Exp= o-e= -2.5 v= 5.5 odds ratio= 0.64 Conf.Int.= P= 0.29 Courtesy of Dr. K. Wheatley

Compare only patients in one trial with patients in the same trial Statistics Obs’d – exp’dVariance Trial 1 (o – e) 1 V 1 Trial 2 (o – e) 2 V 2 Trial 3 (o – e) 3 V 3 All Trials (o – e) T V T Courtesy of Dr. K. Wheatley

Compare only patients in one trial with patients in the same trial Statistics Obs’d – exp’dVariance Trial Trial …… Trial All Trials Odds ratio = % confidence interval: 0.49 to 0.83 P<0.001

Rationale for (quantitative) synthesis of all available evidence Reduction of bias: Comparison of alike with alike –Use of randomized comparison whenever possible –Always within the same trial –Pooling is done by adding trials (not patients) Reduction of imprecision and uncertainty –Particularly important when the effects of interventions are of small to moderate size (e.g. RRR=5-10% or 15-25%) 20% of reduction in a 50% risk of death=avoidance of death in 1 in 10 patients

Effect of random errors Function of the size of the trial Subgroup analysis

No evidence or no evidence of an effect? Absence of evidence of benefit is not evidence of absence of benefit Truly negative trial (evidence of no effect) vs. false-negative trial (no evidence of an effect)

Size of randomized trials in myeloma

Effect of chance: data-dependent subgroup analysis vs. indirect extrapolation of overall analysis Data-dependent subgroup analyses may result to importantly biased conclusions… and should be avoided… Paradoxically, even effects among specific categories of patients may be best assessed indirectly by approximation of overall treatment effect to the patients into a specific category qualitatively –As long as the effect the effect in the specific subgroup is not qualitatively different from the overall effect

Real trial (ISIS-2): EXAGGERATEDLY POSITIVE mortality effect in a subgroup defined only by astrological “birth sign” Astrological “birth sign”Atenolol effect on day 0-1 mortality in acute myocardial infarction Mortality reduction Statistical comparing Atenlol significance with control group(2P) Leo (I.e. born beween71% + 23<0.01 July 24 & August 23) 11 other birth signs Mean 24% E ach > 0.1 (NS) (taken separately) Any birth sign 30% + 10<0.004 (appropriate overall analysis)