Effects of lack of independence in meta-epidemiology Peter Herbison Preventive and Social Medicine University of Otago

The problem Median number of trials in a meta-analysis in the Cochrane Library is 2-3. In spite of this many of these reviews make quite strong recommendations. Are they justified in making these recommendations?

What we wanted to do Used an existing data set that has 65 meta-analyses from 18 systematic reviews that was collected for another purpose Using cumulative meta-analysis we looked at what the answer was after the first three and the first five studies and compared this with the answer from all the studies (“final” answer)

Referees Paper came back from the journal saying that it was a good idea but they were not certain if using multiple outcomes from the same systematic review was reasonable Most similar meta-epidemiology studies only select one outcome from each systematic review This would leave us with only 18 results

Lack of independence I find it hard to imagine that this lack of independence will influence how quickly results settle down Especially since there is often a different mix of studies for the different outcomes One referee suggested a sensitivity analysis using one outcome from each review

Bootstrapping Why just randomly choose one outcome from each review when you can do this repeatedly? –Using strata and size in the bootstrap command This should give some idea whether the lack of independence is important or not

Results Does the confidence interval include the “final” value? EstimateBinomial 95% CI Bootstrap 95% CI After 3 studies 68.8%57.1 – – 80.7 After 5 studies 75.0%64.1 – – 87.9

Results Does the confidence interval overlap with that of the “final” value? EstimateBinomial 95% CI Bootstrap 95% CI After 3 studies 75.0%64.1 – – 83.7 After 5 studies 81.2%71.5 – – 91.7

More traditional meta-epidemiology Use the same data set to see if lack of allocation concealment is associated with bias. Assuming independence –ROR 0.91 (95%CI 0.84 – 0.98) Bootstrap –ROR 0.91 (95%CI 0.83 – 0.99)

Conclusion In this data set at least, lack of independence does not seem to make much difference.