Course: Research in Biomedicine and Health III Seminar 5: Critical assessment of evidence

 EBM steps ◦ Step 1: Formulating questions that can be answered ◦ Step 2: Finding best evidence ◦ Step 3: Quick critical assessment of the evidence ◦ Step 4: Applying evidence ◦ Step 5: Assessing effectiveness and efficiency of the process

3. Critical assessment of evidence 1.Are study results valid? 2.What are the results? 3.Can the results be applied to a concrete patient?

 minimize bias ◦ – of the reviewer, and in the research studies themselves  enhance precision ◦ – by including all the relevant evidence  put results into context ◦ – by examining conflicts and understanding differences  help prioritize research ◦ – by knowing exactly what has been done, how well, and with what findings

Meta-analysis is “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” (Glass, 1976) Other terms: pooled effect / pooled analysis overall effect / summary effect combination of estimates quantitative synthesis

 Each trial is summarised by a measure of effect  Those are combined into a summary estimate of effect, taking into account the amount of information available in each study (bigger studies - more weight)  The overall measure of effect is a weighted average of the results of the individual trials  Important to consider uncertainty of resulting estimate (confidence interval)

 When more than one study evaluated the effect  Where there are no differences in those aspects of the studies which could significantly affect the outcomes.  When the outcome was measured in a similar way.  When the data are available.

TreatedControls (death/No.) (death/No.) Trial A 40 / / 200 Trial B 15 / / 100

TreatedControlsRelative risk (death/No.) (death/No.) (95% CI) Trial A 40 / / (1.14, 2.25) Trial B 15 / / (0.56, 4.01)

TreatedControlsRelative risk (death/No.) (death/No.) (95% CI) Trial A 40 / / (1.14, 2.25) Trial B 15 / / (0.56, 4.01) Total55 / / (0.71,1.40)

Simple vs weigthed average: all studies given equal weight vs. varying importance of trials: more weight if more information (depends mostly on sample size) more information → increased precision Weight of a study depends on: sample size frequency of the event for binary outcomes between-person variability (SD) (for continuous outcome) chosen summary statistic

 Forest plot (blobbogram) 1.0 Favors control Favors treatment Risk ratio Scale (effect measure) Estimate and CI for each study, size proportional to weight Estimate and CI for MA Line of no effect Direction of effect

 Estimate of MA should be interpreted in the context of clinically important effect size.  Confidence interval helps us estimate what might happen in the whole population of interest, not only the sample in the study(ies).

 Because of the error of measurement our results are always just an estimate of the real situation in the population  Confidence interval : Interval in which, with a certain degree of confidence (95%, 99%), is the “true” result in the population  Example: M=20, 95% CI C=76, 99% CI 69-85

 A random interval with endpoints computable from sample data, such that 95 of every 100 such intervals (obtained from similar samples of the same size and from the same population) will include the TRUE (population) value  Also, the range of plausible parameter values indicates precision of the estimate

A random interval with endpoints computable from sample data such that 95 of every 100 such intervals (obtained from similar samples of the same size and from the same population) will include the TRUE (population) value Confidence interval (95%CI)

 Better than 1000 hypothesis tests  Helps us decide whether statistically significant difference is also clinically significant  Example: “the drug lowered diastolic blood pressure in the intervention group by a mean of 17 mmHg (M t1 =99mmHg, M t2 =82mmHg; paired samples t-test, p=0.027)” “the drug lowered diastolic blood pressure in the intervention group by a mean of 17 mmHg (M t1 =99mmHg, M t2 =82mmHg; 95%CI for difference=2-36 mmHg)”

What should be reported in an article on SR? PRISMA statement Check list Flow diagram