Translating Evidence-based Assessment into Educational Practice: Building a DREAM Christie Palladino, MD, MSc; Brittany Bodie, MS; Rachel Elam, ScM; Margo Villarosa*, MS; Lindsey West, PhD; Kirsten Wildermuth*, MS; Lara Stepleman, PhD Education Discovery Institute Georgia Health Sciences University, Augusta, GA *Augusta State University, Augusta, GA Background Results Description, continued Common barriers to highly-effective medical education research include the lack of accessibility to and knowledge of validated assessment instruments. As the AAMC’s peer-reviewed repository and home to hundreds of assessment submissions, MedEdPORTAL serves as a user-friendly location for a publication to help alleviate these barriers. Total Measures n=2,147 Screen shot of the future DREAM website Single Instance/ No Psychometrics PEARLS Single instance with no reported psychometric data n=920 Single instance with reported psychometric data n=366 Objective 42.9% 3% To create DREAM, the Directory and Repository of Educational Assessment Measures, as a collaboration between the GHSU Education Discovery Institute and MedEdPORTAL. To provide educators and medical education researchers with a “one-stop shop” assessment measure database that is accessible to the public. To feature expert analyses as part of Critical Synthesis Packages for each non-proprietary measure. Multiple use with no reported psychometric data n=65 17% Multiple Use/ No Psychometrics PEARLS n=796 Single Instance/ + Psychometrics Publication Description Summary Points CRITICAL SYNTHESIS PACKAGE Author-contributed copy of the ASSESSMENT TOOL Expert CRITICAL ANALYSIS of the measure SUPPLEMENTARY MATERIALS (e.g. scoring guides) THE DREAM PROCESS: Obtain permission from original instrument developers Call for authors to write critical analyses Author submits a Critical Synthesis Package Package is peer-reviewed Accepted package becomes part of the DREAM collection Our results highlight the need for more consistent, high-quality assessment in medical education. The DREAM repository will serve as a publically accessible resource for facilitating the translation of evidence-based assessment into everyday medical education practice. In line with MedEdPORTAL’s expansion in scope, we also plan over time to include instruments developed and/or validated within other health professions education. Be on the lookout for the launch of DREAM in 2013! Methods Step 1: We will invite original developers of PEARLS to provide written permission for use of their intellectual property and to submit copies of their instruments and supplementary materials Step 2: In the spring of 2012, a call will go out, recruiting authors to write the expert critical analysis of each PEARL Steps 3-4: Authors will submit Critical Synthesis Packages to MedEdPORTAL for peer review. Each package will contain an expert critical analysis of the measure, including: Type of Assessment Brief Description/Purpose Development & Psychometrics Additional Studies Supporting Validity and/or Reliability Application to Medical Education and Research Strengths and Weakness Step 5: Accepted packages will form the DREAM collection We performed a two-phased systematic literature search, developed in consultation with two research librarians.** Review of the five most recent years of publications from four key medical education journals; and MeSH (Medical Subject Heading) term and keyword search of articles in PubMed. We included English-language studies that reported an explicit tool or method used to assess medical students or residents. We abstracted information to identify PEARLS: ‘Psychometrically, Evidenced Appearing Repeatedly in a Literature Search.’ PEARLS =reported psychometric data, cited ≥2 in the literature and ≥1 in a medical education sample within the last ten years. http://georgiahealth.edu/medicine/discovery/edi/dream/index.html dream@georgiahealth.edu **We would like to acknowledge: Kathy Davies and Lindsay Blake for their work in developing the literature search Jeff Mastromonico for his work in developing the DREAM submission website Janelle Davis, Deborah Richardson, Andria Thomas, Jeff Campbell and Ruth-Marie Fincher for their assistance in developing the database; and Michael Saleh, Robby Reynolds, and Christopher S. Candler for MedEdPORTAL’s collaborative efforts in designing and building DREAM Presented at the Association of American Medical Colleges Annual Meeting November 5, 2011