1. BACKGROUND AND AIM Website: http://www.abdn.ac.uk/hsru Challenges in conducting a systematic review of the diagnostic accuracy of genetic tests: an example of the genetic diagnosis of familial hypercholesterolaemia REFERENCES CONCLUSIONS METHODS P Sharma 1, G Mowatt 1, F Stewart 1, C Boachie 1, Z Miedzybrodzka 2, W Simpson 4, D Boyers 1,3, M Kilonzo 3, P McNamee 3 1. Health Services Research Unit, University of Aberdeen, 2. Clinical Genetics Centre, University of Aberdeen, 3. Health Economics Research Unit, University of Aberdeen, 4. Clinical Biochemistry Laboratory, NHS Grampian, Aberdeen. CONTACT: Pawana Sharma at p.sharma@abdn.ac.uk ACKNOWLEDGMENTS The availability and uptake of genetic tests is increasing. In 2010 the Aberdeen HTA group were commissioned by the UK National Institute for Health Research Health Technology Assessment (NIHR HTA) Programme to undertake a systematic review of Elucigene FH20 and LIPOchip for the diagnosis of familial hypercholesterolaemia 1 on behalf of the National Institute for Health and Care Excellence (NICE) Diagnostics Assessment Programme. Two additional systematic reviews 2,3 of the diagnostic accuracy of genetic tests were selected for comparison. Six major methodological challenges were identified from the three reviews (Figure 1). A quantitative assessment of clinical validity is presented in Figure 2. 1 ) Descriptive analysis of the issues encountered and measures undertaken to resolve these issues during the conduct of the systematic review 1 (Table 1). 2) Comparative analysis to assess the extent to which the issues faced during this review were also encountered by other reviews. This comprised: Overview of systematic reviews of the diagnostic accuracy of genetic tests Qualitative analysis of the methodological aspects (for example, whether or not meta-analysis was performed, susceptibility to bias, quality of reporting etc.) Quantitative assessment of clinical validity, (for example, by comparing the proportion of studies that reported diagnostic indices such as sensitivity, specificity, and the variability of these results) 1.Sharma P et al. Elucigene FH20 and LIPOchip for the diagnosis of familial hypercholesterolaemia: a systematic review and economic evaluation. Health Technol Assess 2012;16(17). 2.Bryant J et al. A systematic review of the clinical validity and clinical utility of DNA testing for hereditary haemochromatosis type 1 in at-risk populations. J Med Genet 2008;45:513-518. 3.Gerhardus A et al. Diagnostic accuracy of methods for the detection of BRCA1 and BRCA2 mutations: a systematic review. European Journal of Human genetics, 2007;15:619-627. AIM: To explore the methodological challenges involved in undertaking a systematic review of the diagnostic accuracy of genetic tests. Genetic tests are continuing to evolve, reviewers should obtain input from scientific/technical experts from the study design phase and undertake intensive scoping of the review question. The paucity of well designed studies included in the three genetic test reviews made a valid assessment of the tests difficult. Study investigators should design studies to allow calculation of specificity as well as sensitivity where possible. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorates. The systematic review of Elucigene FH20 and LIPOchip was funded by the HTA programme (project number 10/70/01) and published in Health Technol. Assess. 2012;16(17). Further information available at: http://www.hta.ac.uk/project/2450.asp. The views and opinions expressed are those of the authors and do not necessarily reflect those of the Chief Scientist Office, HTA programme, NIHR, NICE, NHS or the Department of Health. We thank the NICE assessment subgroup specialist members and Lara Kemp, Cynthia Fraser and Graeme MacLennan for their support. http://www.hta.ac.uk/project/2450.asp RESULTS - DESCRIPTIVE ANALYSIS Issues encountered Description of the issue Decision made Recommendation suggested Dilemma in selection of studies Applying the original strict selection criteria would have resulted in only 3 studies being included. Selection criteria were relaxed, in consultation with the expert advisory panel. Input from scientific experts versed in the technology should be sought from the design phase. Comprehensiveness of the reference standard The PCSK9 gene, a part of the recognised reference standard, is rare and was discovered in 2003, hence would not have been included in the reference standard prior to this date. Studies using a reference standard that did not include coverage of PCSK9 were included as this was still considered to be sufficiently comprehensive in correctly classifying FH. While considering reference genetic tests, the evolution of gene discovery over time, reflecting developments in genetics, should also be taken into account. Lack of false positive data Only the index test- negatives went on to receive further genetic tests, including the reference standard. Specificity was assumed to be 100% (no false positives). However, these studies were judged to be affected by partial and differential verification biases. Diagnostic measures may not be adequately reported and meta- analysis may not be possible or appropriate. An alternative analysis plan should be considered in advance should this eventuality occur. Meta-analysis not conducted as planned The lack of sufficient diagnostic indices and heterogeneity across studies (e.g., clinical criteria, version of tests used). Forest plots of sensitivity and specificity were used to visualise the heterogeneity amongst the included studies. Limited evidence base for economic modelling It was not possible to provide pooled estimates. Sensitivity was taken from a single study and 100% specificity was assumed. Sensitivity analyses should explore the impact of study choices and assumptions on cost-effectiveness results. Decisions addressing these issues were made in consultation with our project expert advisory panel. Key methodological challenges in conducting a systematic review of the diagnostic accuracy of genetic tests Difficulty adhering to the strict inclusion criteria as defined a priori Lack of valid diagnostic indices Difficulty defining the reference standard Heterogeneity across studies Inability to perform meta-analysis Evidence gap in terms of both quantity and quality Figure 1: Key methodological challenges in conducting a systematic review of the diagnostic accuracy of genetic tests Figure 2: Diagnostic indices and variability of these results as reported by the studies included in three reviews RESULTS - QUALITATIVE ANALYSIS RESULTS - QUANTITATIVE ASSESSMENT OF CLINICAL VALIDITY RESULTS - COMPARATIVE ANALYSIS Table 1: Examples of issues encountered, measures undertaken to resolve the issues and suggested recommendations