1. Translating Evidence-based Assessment into Educational Practice:
Building a DREAM
Christie Palladino, MD, MSc; Brittany Bodie, MS; Rachel Elam, ScM; Margo Villarosa*, MS; Lindsey West, PhD; Kirsten Wildermuth*, MS; Lara Stepleman, PhD
Education Discovery Institute
Georgia Health Sciences University, Augusta, GA
*Augusta State University, Augusta, GA
Background
Results
Description, continued
Common barriers to highly-effective medical education research include the lack of accessibility to and knowledge of validated assessment instruments.
As the AAMC’s peer-reviewed repository and home to hundreds of assessment submissions, MedEdPORTAL serves as a user-friendly location for a publication to help alleviate these barriers.
Total Measures
n=2,147
Screen shot of the future DREAM website
Single Instance/
No Psychometrics
PEARLS
Single instance with no reported psychometric data
n=920
Single instance with reported psychometric data
n=366
Objective
42.9% 3%
To create DREAM, the Directory and Repository of Educational Assessment Measures, as a collaboration between the GHSU Education Discovery Institute and MedEdPORTAL.
To provide educators and medical education researchers with a “one-stop shop” assessment measure database that is accessible to the public.
To feature expert analyses as part of Critical Synthesis Packages for each non-proprietary measure.
Multiple use with no reported psychometric data
n=65
17%
Multiple Use/
No Psychometrics
PEARLS
n=796
Single Instance/
+ Psychometrics
Publication Description
Summary Points
CRITICAL SYNTHESIS PACKAGE
Author-contributed copy of the
ASSESSMENT TOOL
Expert
CRITICAL ANALYSIS
of the measure
SUPPLEMENTARY MATERIALS
(e.g. scoring guides)
THE DREAM PROCESS:
Obtain permission from original instrument developers
Call for authors to write critical analyses
Author submits a Critical Synthesis Package
Package is peer-reviewed
Accepted package becomes part of the DREAM collection
Our results highlight the need for more consistent, high-quality assessment in medical education.
The DREAM repository will serve as a publically accessible resource for facilitating the translation of evidence-based assessment into everyday medical education practice.
In line with MedEdPORTAL’s expansion in scope, we also plan over time to include instruments developed and/or validated within other health professions education.
Be on the lookout for the launch of DREAM in 2013!
Methods
Step 1: We will invite original developers of PEARLS to provide written permission for use of their intellectual property and to submit copies of their instruments and supplementary materials
Step 2: In the spring of 2012, a call will go out, recruiting authors to write the expert critical analysis of each PEARL
Steps 3-4: Authors will submit Critical Synthesis Packages to MedEdPORTAL for peer review.
Each package will contain an expert critical analysis of the measure, including:
Type of Assessment
Brief Description/Purpose
Development & Psychometrics
Additional Studies Supporting Validity and/or Reliability
Application to Medical Education and Research
Strengths and Weakness
Step 5: Accepted packages will form the DREAM collection
We performed a two-phased systematic literature search, developed in consultation with two research librarians.**
Review of the five most recent years of publications from four key medical education journals; and
MeSH (Medical Subject Heading) term and keyword search of articles in PubMed.
We included English-language studies that reported an explicit tool or method used to assess medical students or residents.
We abstracted information to identify PEARLS: ‘Psychometrically, Evidenced Appearing Repeatedly in a Literature Search.’
PEARLS =reported psychometric data, cited ≥2 in the literature and ≥1 in a medical education sample within the last ten years.
**We would like to acknowledge:
Kathy Davies and Lindsay Blake for their work in developing the literature search
Jeff Mastromonico for his work in developing the DREAM submission website
Janelle Davis, Deborah Richardson, Andria Thomas, Jeff Campbell and Ruth-Marie Fincher for their assistance in developing the database; and
Michael Saleh, Robby Reynolds, and Christopher S. Candler for MedEdPORTAL’s collaborative efforts in designing and building DREAM
Presented at the Association of American Medical Colleges Annual Meeting November 5, 2011