Completeness of reporting of studies of diagnostic accuracy in Cardiology journals: a STARD assessment Key words: STARD checklist, Cardiology, Diagnostic accuracy Class 6: Ana Borges, Ana Torres, André Tojal, Célia Soares, Francisca Trigo, Francisco Coutinho, Joana Ribeiro, José Fernandes, Pedro Vasconcelos, Raquel Ferreira, Sílvia Madureira Supervisors: Altamiro da Costa Pereira, MD, PhD; Filipa Almeida, MD Introduction to Medicine; Porto Faculty of Medicine 2008

ObjectivesMethodsAcknowledgmentsResults Introduction Conclusion

Introduction ObjectivesMethodsAcknowledgmentsResults Accurate diagnosis in Cardiology is crucial for clinical intervention and is increasingly important as the number of validated treatment for specific conditions increases. The use of systematic literature review to inform evidence based practice in diagnostics is rapidly expanding. Studies of the new diagnostic tests, and their development, are reported in scientific articles. (1) (1) - Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW. Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative. Ann Intern Med. 2003; 138:40-44 Conclusion

Introduction ObjectivesMethods Acknowledgments Results Current cardiology practice relies on diagnostic tests using sophisticated technologies that are constantly evolving. Diagnostic accuracy studies are required to validate new diagnostic tests before they are introduced into clinical practice. These studies typically report sensitivity and specificity, likelihood ratios, diagnostic odds ratio, or area under a receiver operating characteristics (ROC) curve as measures of diagnostic performance. This information enables a clinician to make judgments regarding the potential utility of new tests. (2) (2) Deeks J. Systematic reviews in health care: systematic reviews of diagnostic tests, BMJ 2001; 44: Conclusion

Introduction ObjectivesMethods Acknowledgments Results A rigorous evaluation process of diagnostic tests before introduction into clinical practice could not only reduce the number of unwanted clinical consequences related to misleading estimates of test accuracy, but also limit health care costs by preventing unnecessary testing. Several systematic reviews (1995, Reid et al) (3) have emphasized the poor quality of reporting in diagnostic accuracy studies. This poor reporting hampers an adequate judgment of both the internal and the external validity of a study. (3) Scholten RJ, Deville WL, Opstelten W, Bijl D, van der Plas CG, Bouter LM. The accuracy of physical diagnostic tests for assessing meniscal lesions of the knee: a meta-analysis. J Fam Pract. 2001; 50:938–944. Conclusion

Introduction ObjectivesMethods Acknowledgments Results 1999 In 1999, in Rome, the Cochrane Diagnostic and Screening Test Methods Working group discussed the low methodological quality and sub-standard reporting of diagnostic test evaluations, that resulted in the intent of developing a checklist of items that should be included in those reports The STARD steering committee started with an extensive search to identify publications on the conduct and reporting of diagnostic studies up to July (4) 2003 In January 2003, after the discussion based on the search done, we have the first official version of the STARD, which was published simultaneously in eight medical journals. History of the STARD (4) Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003 Conclusion

Introduction ObjectivesMethods Acknowledgments Results STARD CHECKLIST Objective: The aim of the Standards for Reporting of Diagnostic Accuracy (STARD) initiative is to improve the quality of reporting of studies on diagnostic accuracy.How: Using a checklist with 25-item and a flow diagram, which provides information about the method of patient recruitment, the order of test execution and the numbers of patients undergoing the test under evaluation, the reference standard or both. (5)Advantage: Complete and accurate reporting allows the reader to detect the potential for bias in the study and to evaluate the general ability of the results. (5) Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA. 1995; 274:645–651. Conclusion

Objectives Methods Acknowledgments Results Cardiology is the area with the most published articles; This is the first study that analyses how reports on diagnostic accuracy in cardiology comply with the STARD checklist; Introduction Importance of our analysis Conclusion

Introduction Objectives Methods Acknowledgments Results To assess the quality of reporting of studies on diagnostic tests in cardiology journals, based on the STARD checklist. Main objective Secondary objectives To analyse the quality of diagnostic tests studies in the course of the years; To compare the differences between the articles published before and after the creation of the STARD checklist; To evaluate the influence of the journals’ impact factor on the quality of the articles which report diagnostic tests in cardiology journals. Conclusion

Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Introduction ObjectivesMethods Acknowledgments Results Standard query to find articles that report studies of diagnostic accuracy (in PubMed) (((((((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields)) OR (((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All fields]) OR "roc and"[All fields]) OR "roc area"[All fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields])) OR "likelihood ratio"[All Fields]) Devillé W. Conducting systematic reviews of diagnostic studies; didactic guidelines. Medical research methodology Conclusion

Introduction ObjectivesMethods Acknowledgments Results Search the journals Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Included all journals classified as “cardiac and cardiovascular systems” in the ISI web of knowledge except those who are not found on pubmed. Included all 71 journals found on Pubmed Conclusion

Introduction Objectives Methods AcknowledgmentsResults Final query (after addition of terms to specify the search to the journals) Articles’ search Articles’ selection Articles’ evaluation Statistical analyze (((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR ("likelihood ratio"[All Fields])) AND (Circulation [Journal] OR Circulation Research [Journal] OR Journal Of The American College Of Cardiology [Journal] OR European Heart Journal [Journal] OR Cardiovascular Research [Journal] OR Journal Of Molecular And Cellular Cardiology [Journal] OR American Journal Of Physiology-Heart And Circulatory Physiology [Journal] OR Journal Of Thoracic And Cardiovascular Surgery [Journal] OR Trends In Cardiovascular Medicine [Journal] OR American Journal Of Cardiology [Journal] OR Journal Of Heart And Lung Transplantation [Journal] OR Chest [Journal] OR American Heart Journal [Journal] OR Journal Of Cardiovascular Pharmacology [Journal] OR Progress In Cardiovascular Diseases [Journal] OR Journal Of Cardiovascular Magnetic Resonance [Journal] OR Heart [Journal] OR Journal Of Nuclear Cardiology [Journal] OR Annals Of Thoracic Surgery [Journal] OR Journal Of The American Society Of Echocardiography [Journal] OR Current Opinion In Cardiology [Journal] OR Current Problems In Cardiology [Journal] OR Basic Research In Cardiology [Journal] OR Catheterization And Cardiovascular Interventions [Journal] OR Respiratory Medicine [Journal] OR Canadian Journal Of Cardiology [Journal] OR European Journal Of Cardio-Thoracic Surgery [Journal] OR European Journal Of Heart Failure [Journal] OR Journal Of Heart Valve Disease [Journal] OR Clinical Cardiology [Journal] OR Cardiovascular And Interventional Radiology [Journal] OR International Journal Of Cardiology [Journal] OR Herz [Journal] OR Cardiovascular Drugs And Therapy [Journal] OR Journal Of Cardiothoracic And Vascular Anesthesia [Journal] OR Cardiovascular Drug Reviews [Journal] OR Zeitschrift Fur Kardiologie [Journal] OR Pediatric Cardiology [Journal] OR Cardiovascular Surgery [Journal] OR Thoracic And Cardiovascular Surgeon [Journal] OR Journal Of Interventional Cardiac Electrophysiology [Journal] OR Revista Espanola De Cardiologia [Journal] OR Cardiology [Journal] OR Heart & Lung [Journal] OR Cardiology In The Young [Journal] OR Heart And Vessels [Journal] OR Journal Of Electrocardiology [Journal] OR Journal Of Cardiovascular Surgery [Journal] OR International Journal Of Cardiac Imaging [Journal] OR Texas Heart Institute Journal [Journal] OR Scandinavian Cardiovascular Journal [Journal] OR Acta Cardiologica [Journal] OR Europace [Journal] OR Archives Des Maladies Du Coeur Et Des Vaisseaux [Journal] OR Cardiovascular Pathology [Journal] OR Japanese Heart Journal [Journal] OR Journal Of Cardiac Surgery [Journal] OR Kardiologiya [Journal]) Conclusion

Introduction ObjectivesMethods Acknowledgments Results Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Random sample of 100 articles stratified by years Sample of 10 articles per year Years included: - Until 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, From all articles retrieved by the query Conclusion

Introduction Objectives Acknowledgments Results Methods Reading and analyzing the abstract Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Inclusion Inclusion - to report a study of diagnostic accuracy - the diagnostic test(s) is (are) used in cardiology - the full text is available The abstract is read and analyzed with the inclusion/exclusion criteria Conclusion

Introduction Objectives Acknowledgments Results Methods Reading and analyzing the abstract Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Exclusion Exclusion - Is a clinical trial, a letter, an editorial, a practice guideline, a meta- analysis or a review -The article could not be found in the Internet, the facilities of the School of Medicine or the local libraries -The article is written in languages other than English, Portuguese, French or Spanish - The article found is not complete Conclusion

Introduction Objectives Acknowledgments ResultsMethods Revision Articles’ search Articles’ selection Articles’ evaluation Statistical analyze If excluded: - the reason of exclusion is registered If doubted: - the abstract is reanalyzed by other selector When disagreement or doubt occur, the article will be discussed by both selectors. When no decision could be achieved, a third selector will be consulted When this situation occurs, PMID of each opened article will be registered Conclusion

Introduction Objectives Acknowledgments ResultsMethods Final samples Articles’ search Articles’ selection Articles’ evaluation Statistical analyze 2007 → 139 articles 2006 → 136 articles 2005 → 125 articles 2004 → 132 articles 2003 → 125 articles 2002 → 147 articles 2001 → 112 articles 2000 → 172 articles 1999 → 112 articles Until 1998 → 1585 articles Conclusion

Introduction Objectives Acknowledgments ResultsMethods Final samples Articles’ search Articles’ selection Articles’ evaluation Statistical analyze TOTAL → 2785 articles Conclusion

IntroductionObjectivesAcknowledgmentsResultsMethods Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Flowchart

Introduction Objectives Acknowledgments ResultsMethods Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Gantt Chart Conclusion

Articles’ search Articles’ selection Articles’ evaluation Statistical analyze ObjectivesResultsMethods Selection and TopicItemDescribe TITLE/ABSTRACT/ KEYWORDS 1 Implied The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity') Implied INTRODUCTION 2 The main objectives are present The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups The main objectives are present METHODS Participants 3 It has inclusion and exclusion criteria The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected It has inclusion and exclusion criteria 4 If it has two of the three Participant recruitment: was this based on presenting symptoms, results from previous tests, or the fact that the participants had received the index test(s) or the reference standard? If it has two of the three 5 If 3 and 4 have been marked Participant sampling: was this a consecutive series of patients defined by selection criteria in (3) and (4)? If not specify how patients were further selected. If 3 and 4 have been marked 6 Data collection: were the participants identified and data collected before the index test(s) and reference standards were performed (prospective study) or after (retrospective study)? If it says before ou after / implied Reference standard 7 Nor that it doesn’t have rational analysis The reference standard and its rationale Nor that it doesn’t have rational analysis Test methods 8 Technical specification of material and methods involved including how and when measurements were taken, and/or cite references for index test(s) and reference standard Nor that it doesn’t specify how and where 9 If it has the two topics Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference standard If it has the two topics 10 If it has at least two of the three The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference standard If it has at least two of the three 11 If it has the first topic or both Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other test(s) and describe any information available to them If it has the first topic or both Statistical methods 12 Only if it has both Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals) Only if it has both 13 If yes / if it has been done Methods for calculating test reproducibility, if done If yes / if it has been done STARD CHECKLIST Each item values 1 point Several criteria was defined for the STAR application

Articles’ search Articles’ selection Articles’ evaluation Statistical analyze ObjectivesResultsMethods Section and topicItemDescribe RESULTS Participants 14 If it has everything or the duration time When study was done, including beginning and ending dates of recruitment If it has everything or the duration time 15 f it presents at least one of each Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s), comorbidity, current treatment(s), recruitment center) I f it presents at least one of each 16 If it has everything, even it doesn’t have the diagram How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the reference standard; describe why participants failed to receive either test (a flow diagram is strongly recommended) If it has everything, even it doesn’t have the diagram Reference standard 17 If it has everything Time interval and any treatment administered between index and reference standard If it has everything 18 If it has the first topic or both Distribution of severity of disease (define criteria) in those with the target condition; describe other diagnoses in participants without the target condition If it has the first topic or both Test results 19 it has the first topic or both, in case the second is verified A cross tabulation of the results of the index test(s) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard If it has the first topic or both, in case the second is verified 20 If it has the item Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result and how they were handled If it has the item 21 If it has both Adverse events of index test(s) and reference standard If it has both Estimation 22 If it has two of the three Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals) If it has two of the three 23 If it has everything Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done If it has everything 24 If it has the item and it’s done Measures of test reproducibility, if done If it has the item and it’s done DISCUSSION 25 If it has the item The clinical applicability of the study findings If it has the item STARD CHECKLIST Each item values 1 point Several criteria was defined for the STAR application

Introduction Objectives Acknowledgments ResultsMethods Evaluators – Distribution for year Article’s components - Title, - Abstract, - Keywords - Introduction - Methods - Results - Discussion One evaluator for each year Every article´s components are seen for the same evaluator 10 articles for each evaluator Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Conclusion

Introduction Objectives Acknowledgments ResultsMethods Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Final score of each article Average score of the articles for each group (year of publication, impact factor) Interobserver variability The status of each STARD item in each year Points to analyse: Conclusion

Introduction Objectives Acknowledgments Results Methods Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Statistical Analyse Conclusion

Introduction Objectives Acknowledgments Results Methods Final Selection States From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results Methods Reasons for the Exclusion From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results Methods Final Classification of the Items From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results Methods ICC for each Item of the List From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results Methods Evolution of the Articles’ Quality From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results Methods The Impact Factor of the Journal From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results MethodsConclusion. S. Some items of evaluation are rigorously asked for in a section, but in the articles, several times, they appear in another one;.. Because some STARD items are potentially affected by the subjective judgment of the reviewers due to differences in interpretation of the STARD, disagreement arises; About the STARD CHECKLIST – Evaluators’ point of view

Introduction Objectives Acknowledgments Results Methods Relation Between the Articles’ Score and the Impact Factor of the Journal From Articles’ Selection From Articles’ Evaluation Conclusion

Introduction Objectives Acknowledgments Results MethodsConclusion.. The data of the participants (age, sex, symptoms,…) and their dates of recruitment should be included in the methods;.. The list has 25 items, but it’s possible each one can have more then one aspect to evaluate, which creates confusion in the classification to attribute. About the STARD CHECKLIST – Evaluators’ point of view

Introduction Objectives Acknowledgments Results Methods Expected Results vs Obtained Results Expected As we can see in other medical areas where this project had been done, such as obstetrics and gynecology and others, we expect that this checklist had had impact on the articles written after it as well as that they had improve its qualities due to STARD Checklist. Conclusion

Introduction Objectives Acknowledgments Results Methods Expected Results vs Obtained Results Obtained.. The differences found in the articles were nothing to do with variances observed between groups, we mean the year of publish is not related with the improvement of the articles (p<0,005)... Therefore we assume that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy and so we suppose that the improve in this kind of articles is due to the progress of knowledge along the years and not related with the creation of this checklist, as shown in the graphic... We come to conclusion that there is no correlation between them, since we reach a p value equal to 0,210, that is not significant and by the way that we can declare that the year of publish or the improve verify in the articles does not interfere with the impact factor of the journals. Conclusion

Introduction Objectives Acknowledgments ResultsMethods Conclusion Our study shows that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy. Conclusion

IntroductionObjectivesAcknowledgments ResultsMethods Altamiro da Costa Pereira, MD,PhD Filipa Almeida, MD, PhD Cristina Santos, MD, PhD Luís Azevedo, MD, PhD Acknowledgments