1. Author Classification of ARVO Conference Abstracts as Controlled Clinical Trials Roberta W. Scherer 1, Pamela C. Sieving 2, and Kay Dickersin 1, Johns Hopkins Bloomberg School of Public Health 1, NIH Library 2 Only about 60% of controlled clinical trial (CTs) presented at scientific meetings and published as abstracts in conference proceedings are published in full (Scherer, 2005). Accordingly, it is important to handsearch conference proceedings to ensure comprehensive trial identification for systematic reviews. The annual meeting of the Association for Research in Vision and Ophthalmology (ARVO) is an important source of reports of clinical trials in the field of vision yet only a small proportion of abstracts describe CTs (e.g., 214/5,061 (4%) 2001 ARVO abstracts were identified as CTs by a handsearcher.) Handsearching ARVO abstracts is thus not an efficient method of identifying CTs. BACKGROUND METHODS OBJECTIVE To test whether abstract authors could contribute to efficient identification of controlled clinical trial (CT) reports published in ARVO proceedings. At our request, the ARVO conference organizers included a question and accompanying definition on the electronic abstract submission form for 2004: “Is the research presented in your abstract a human clinical trial? [Yes] [No].” In 2005, the question was modified to “Is the research presented in your abstract a human controlled clinical trial? [Yes] [No]. For both years, the definition, obtainable via hypertext of the blue underlined text, was “A planned study in humans designed to assess the efficacy and/or safety of one or more test interventions by comparing outcomes in individuals assigned the test intervention(s) with those receiving no intervention or a comparison intervention, and where individuals in all groups are enrolled, treated, and followed concurrently.” One of us (RS), masked to authors’ CT classification, handsearched 2004 and 2005 ARVO abstracts online. A gold standard of abstracts classified as CTs was determined by examination of the two sets of abstracts: 1) abstracts classified as CTs by the author and 2) those classified as CTs by the hand-searcher as CTs. Each abstract in the gold standard set was further categorized as a randomized controlled trial (RCT) or controlled clinical trial (CCT), using Cochrane Collaboration definitions. We evaluated the author and handsearcher classifications against the gold standard, and explored factors associated with author misclassification. 2004 More than one reason applied in some abstracts. 2005 RESULTS 295 of 5,610 abstracts from the 2004 ARVO meeting, and 329 of 5,732 abstracts from the 2005 ARVO meeting comprised the gold standard of abstracts describing CTs. Abstract authors checked the box indicating that they considered the abstract to describe a human CT in 634 abstracts in 2004, and 485 in 2005. The most common reason for misclassification (false positives) by abstract authors was that there was no comparison group, ie., the study was a prospective series with everyone receiving the same treatment. The second most common reason was participants were assigned to a group using some participant characteristic (such as having a disease or not). Typically, these were case control studies. Classification as Clinical Trial by Author In 2004, authors correctly classified CTs for 229/295 CT abstracts, (sensitivity = 77.6%). However, authors failed to recognize 66/295 CTs (22.4%) and mis-classified 405 studies as CTs that were not (positive predictive value = 229/634, 36.1%). In 2005, sensitivity was lower than for 2004 (70.5%, 232/329 CT abstracts) and there were more false negatives (97/329; 29.5%) but the positive predictive value was higher (47.8% (232/485 abstracts)). Gold Standard By AuthorYes NoTotal Yes232 253 485 No 9751505247 Total32954035732 2005 Gold Standard By AuthorYes NoTotal Yes229 405 634 No 6649104976 Total29553155610 2004 Gold Standard By hand searcherYes NoTotal Yes269 0 269 No 2653155341 Total29553155610 2004 Classification as Clinical Trial by Handsearcher Of the 295 abstracts in the gold standard, the handsearcher classified 91.1% (269/295) as a CT in 2004 and 93.3% (307/329) in 2005. Gold Standard By hand searcher Yes NoTotal Yes307 0 307 No 2254035403 Total32954035732 2005 Reasons* for Misclassification of Abstracts as a CT by the Abstract Author COMMENT AND FUTURE DIRECTIONS Comment: Earlier studies showed handsearching of the printed ARVO meeting abstracts requires a considerable time investment (about 45 hours per annum). Our project to develop a valid and reliable system for author classification of CTs and electronic searching has the potential to shorten this time. Because we found a large proportion of abstracts misclassifed (false positives) in 2004, the 2005 ARVO abstract submission information was modified by changing the study design question to ask “Is the research presented in your abstract a human controlled clinical trial?” Failure to identify abstracts as human CTs (false negatives) is of greater concern than false positives, as omission is less easily addressed than overselection in the handsearching process. The proportion of false negatives increased, ifanything, between 2004 and 2005. Asking authors to classify abstracts identifies a small number of abstracts (26 and 22, respectively) that would be missed by a handsearcher. Some abstracts describing CTs may have been missed by both the author and the handsearcher; this is a limitation of the study. Future directions We plan to explore: How best to ensure capture of CTs not so identified by authors (false negatives). Identification of a concise set of text words and terms with high sensitivity for identification of CTs in electronic-only meeting abstracts. These will be validated using proceedings from years other than 2004 or 2005. Keywords and abstract classification codes provided with the ARVO abstracts, to determine their usefulness in searching. Author Identification of Clinical Trials by Design Sub-classification Most of the differences in classification between authors and handsearchers are among abstracts categorized as controlled clinical trials (CCTs). Authors identified 57.5% abstracts categorized as CCTs in 2004 while the handsearcher identified 86.8% (92/108). In 2005, authors identified 46.2% (58/119), while the handsearcher identified 84.0% (100/119) of CCTs. Authors may more readily understand that randomized studies (RCTs) are trials compared with CCTs. A study is a CCT if it could possibly be a randomized or quasi-randomized study, so another explanation is that some trials are incorrectly classified as trials when they are not by the handsearcher. CONCLUSIONS Neither the author checkox nor handsearching identified all CT abstracts. Including the checkbox on the abstract submission form identifies a small groups of abstracts missed by handsearching. RCTs were more readily identified as clinical trials by both authors or hand- searchers compared to CCTs (quasi-randomized or possible randomized trials). 20042005