Assessing the student experience: creating a culture of assessment in undergraduate and professional education.” Presented at the Annual NEASC Meeting December 9, 2016 Deborah Harmon Hines, PhD Vice Provost for School Services University of Massachusetts Medical School

UMMS Competencies for Medical Education We strives to produce graduates who will become caring healers both by assuring that they possess the requisite knowledge and skills and by strengthening their natural talents and desire to care for others. The multiple roles of the physician as healer, and the associated competencies that graduating medical students must demonstrate, form the basis for a new way of organizing what is taught, how it is taught and the methods for evaluating student performance at UMMS. They embody the Medical School’s educational philosophy and the distinctive attributes of its faculty and students. These six attributes or roles, which form the cornerstone for a redefining of the Medical School’s educational objectives, are Physician as . . . Professional, Scientist, Communicator, Clinical Problem Solver, Patient & Community Advocate, and Person.

Student Assessment Begins with performance on internal/course examinations Evaluations also done of small group work Foundations of Medicine 1&2 (Y1&2) on CR/NCR Monthly Evaluation Board meetings Core Clinical Experiences (Y3) and Advanced Studies (Y4) on BEP, EP, AEP, O, F, I Clinical courses are also assessed using OSCEs (objective structured clinical examinations) Monthly Evaluation Boards meetings Both Academic Evaluation Board meetings attended by ALL course coordinators, Associate and Assistant Deans for SA, FA, Registrar and Learning Specialist, 2016 NEASC ANNUAL MEETING

Continues with Student Assessment of lectures, lecturers and courses 25 students are randomly selected to assess each lecture and lecturer Information compiled by IREA* and given to Course Coordinators and individual faculty members IREA = Institutional Research, Evaluation and Assessment 2016 NEASC ANNUAL MEETING

Student Performance on Shelf and USMLE “Shelf” exams are provided by the US Medical Licensure Exam (USMLE) in subject areas USMLE is administered in three steps Step 1, at the end of the FOM2 Step 2, at the end of CCE Clinical skills (2CS) Clinical knowledge (2CK) Needed for conditional license to practice Step 3, between residency Y1-Y2 Needed for full licensure Outcomes normed against all who take each section of the USMLE 2016 NEASC ANNUAL MEETING

All topics, titles, key words and placed in the curriculum data base Able to see where and when a topic, title or key word is in the curriculum Allows adjustments for too much redundancy or for missing information 2016 NEASC ANNUAL MEETING

Graduate Questionnaire (GC) Administered by the AAMC Each medical school graduate is given this questionnaire to assess the entire medical school experience Covers: didactic information in all subject areas and clinical care. i.e. Do you feel you learned what was needed to provide competent care about HIV/AIDS? How to treat LGBTQ patients? Also covers: appropriate treatment of learners, FA, discrimination, etc. Were you ever made to feel belittled? Was FA easy to access? Were all of your questions answered? Normed against all medical schools. UMMS is above the mean on 95% of the questions. We still have work to do. 2016 NEASC ANNUAL MEETING

Residency Program Director’s Questionnaire Administered by the AAMC Given to Residency Program Directors each year. Asked to evaluate preparation of each resident and correlated with the resident’s medical school. 2016 NEASC ANNUAL MEETING

Student Support Associate Dean for Student Affairs Collects all grades and presents to Academic Evaluation Boards Assistant Dean for Student Advising assigns each student to a faculty advisor Learning Community House mentors Resident Learning Specialist Course review sessions USMLE Review Sessions Learning Communities Physician mentors Peer mentors Upperclassmen mentors Student ADA services for those who need it 2016 ANNUAL NEASC MEETING

Final thoughts on 360 degree evaluation This system of evaluation has taken years to put into place. It was built one ‘brick’ at a time. The cost/individual and the impact on patients’ lives made a compelling rationale for this system of evaluation. Student Support is key. Internal assessments - Of students be examinations and OSCEs - Of courses and faculty by students - Curriculum data base - Everyone must be at the table External assessments - Shelf exams - USMLE - Graduate Questionnaire - Residency Program Directory Questionnaire