Session #2 Fairness and the New MCAT Exam

AAMC Admissions Hub www.aamc.org/admissions Resources on Fairness FACTS data on MCAT Scores (aamc.org/facts) Using MCAT Data in 2016 Medical Student Selection “Do Racial and Ethnic Group Differences in Performance on the MCAT Exam Reflect Test Bias” (Academic Medicine, May 2013) Khan Academy collection and other free and low cost test preparation resources

Learning objectives By the end of this session, you will: Know how fairness was key to the design and development of the new exam Learn about local efforts to help students prepare for the new exam

Refer to information in this guide during the session

Fairness played a critical role in blueprint development

Fairness played a critical role in blueprint development Balance academic competencies from the natural, behavioral, and social sciences Test psychology and sociology concepts like discrimination, stereotype threat, and socio-economic inequalities Increase attention to population health, studies of diverse cultures, ethics, philosophy Test concepts covered widely in undergraduate science courses

Fairness played a critical role in blueprint development Longer sections that allow medical schools to consider applicants’ strengths and weaknesses in relation to their curricula More working time per question, allowing test takers to work through the information presented in the passage, think about the question, and decide on the best answer

Fairness played a critical role in developing test questions

We used tried-and-true procedures to develop the new questions, and some new ones as well Editorial review Technical review “Summits” with experts Bias and sensitivity review “Field testing” – try out questions to see how they work

Addressing Fairness when Developing Questions for the Psychological, Social, and Biological Foundations of Behavior Section

Potentially sensitive content Some content in the Psychological, Social, and Biological Foundations of Behavior Section is potentially sensitive Prejudice and Bias Discrimination Health Disparities Healthcare Disparities

Gathering feedback on potentially sensitive content Does potentially sensitive Psychological, Social, and Biological Foundations of Behavior section content interfere with the performance of examinees from underrepresented minority groups by distracting them or slowing them down?

Methods of gathering feedback Focus Group with Medical Students and MCAT Exam Editors Review of Responses to PSBB Study Post-administration Survey Interviews with 2013 MCAT Examinees

Results Very few individuals identified the content as distracting Individuals underrepresented in medicine were no more likely to identify the content as distracting than others In general, participants noted that topics such as health disparities are relevant to medical school and important for future medical students to know

Monitoring Fairness Data During Field Testing

New test questions were tried out on examinees in 2013 and 2014 Examinees volunteered to take a 32-item “trial section” at the end of the test Volunteers who showed a “good faith effort” received $30 and feedback on the new questions Data from the trial section were used to: Identify questions with good statistical properties Highlight any opportunities for improved item development

We monitored the field test data for fairness Attended a minority-serving institution or an institution with no pre-health advisor Received fee assistance from the AAMC Males vs. females Race/ethnicity (white, Black, Latino)

Results More than 113k students tried out new questions Participation rates were high, and the vast majority of examinees showed a good faith effort The participants were similar demographically to the population of examinees The high data quality helped build a large item bank for the new MCAT exam Differences in average performance were comparable to the differences on the old MCAT exam

After Lunch, You’ll See Early Data from the April and May 2015 Examinations

Fairness played a critical role in developing the score scales for the new MCAT exam

Developing the Score Scales with Fairness in Mind

Many data sources influenced the design of the new score scales Surveys of admissions officers on the use of MCAT scores and other applicant data in admissions Acceptance rates for applicants with different ranges of MCAT scores and undergraduate GPAs Graduation rates for medical students with different ranges of MCAT scores and undergraduate GPAs

Medical schools use academic metrics, experiences, and attributes holistically , and demographic data. 3

Acceptance Data

44% of applicants received one or more acceptances Percentages of 2012-2014 applicants admitted into one or more medical schools 44% of applicants received one or more acceptances 4

Percentages of 2012-2014 applicants admitted into one or more medical schools 9% of applicants with GPAs above 3.8 and MCAT scores above 38 are not admitted into any medical schools 4

Percentages of 2012-2014 applicants admitted into one or more medical schools 8% of applicants with GPAs 3.00 - 3.19 and MCAT scores 21-23 are admitted into at least one medical school 4

Medical Student Performance

The vast majority of medical students graduate in four years 87% of medical students graduate in 4 years 44

Medical students with a wide range MCAT scores and UGPAs graduated in 4 years 81% of medical students entering with MCAT scores between 24-26 graduate in 4 years 44

A new exam means new score scales Decision Rules No overlap Emphasis on the center of the scale

14

After Lunch, You’ll Have a Chance to Work with the New Score Reports

Our Research on Fairness is Just Beginning

MCAT validity research The MCAT Validity Committee is responsible for evaluating the new exam and how well it: Supports diversity and is fair Predicts medical students’ academic success Helps admissions officers do their work

Validity committee members Catherine Lucey, M.D. (Chair) Aaron Saguil, M.D., LTC (Vice-chair) Ngozi Anachebe, M.D. Ruth Bingham, Ph.D. Barbara Beckman, Ph.D. Kevin Busche, M.D. Victoria Cannon Steven Case, Ph.D. Deborah Castellano, M.S. Jerry Clark, Ph.D., M.B.A. Julie A. Chanatry, Ph.D. Daniel M. Clinchot, M.D. Liesel Copeland, Ph.D. Hallen Chung, M.A. Martha L. Elks, M.D. William Gilliland, M.D. Jorge A. Girotti, Ph.D., M.A. Joshua Hanson, M.D., M.P.H. Brandon Hunter Loretta Jackson-Williams, M.D. David Jones, Ph.D. R. Stephen Manuel, PhD. Stephanie C. McClure, M.D., FACP Janet McHugh 24 37

Validity committee members Stephen Nicholas, M.D. Wanda Parsons, M.D., FCFP Boyd F. Richards, Ph.D. Stuart Slavin, M.D. Jenna Smith, Ph.D. Aubrie Swan Sein, Ph.D., Ed.M. Doug Taylor Carol Terregino, M.D. Ian W. Walker, M.D. L. James (LJ) Willmore, M.D. Robert A. Witzburg, M.D. David Wofsy, M.D. 24 38

The MCAT validity research will evaluate a number of questions, for example: Diversity and fairness Are differences in average scores for test takers in different racial/ethnic groups smaller or larger than differences on the old exam? Academic preparation Will more humanities and psychology majors take the new exam? Admissions decision making How are medical school admissions committees using scores from the new exam in admissions? Predicting medical student performance Will scores from the new exam predict academic performance throughout medical school?

Two working groups are studying fairness issues Questions Access to preparation resources Access to courses Barriers to success Changes to diversity Other People Pre-health advisors Admissions Diversity affairs Other Data MCAT registration PMQ AMCAS applications Other Change to test Changes to preparation Potential differences in access Potential increases in group differences

We will report validity data as they become available Applications for the 2016 class sent to medical schools Predictive validity results for M1 students reported at AAMC Annual Meeting Predictive validity results for M1, M2 students reported at AAMC Annual Meeting MCAT2015 launches MCAT session at the PDC, preliminary predictive validity results for PSBB 2016 class matriculates Acceptance data reported for 2016 class Acceptance data reported for 2017 class June 2015 Nov. 2017 Nov. 2018 April 2015 July 2015 Aug 2016 Jan. 2017 Jan. 2018 25

Fairness played a critical role in developing preparation resources for aspiring physicians

Helping Students Prepare Understand Practice Study

Helping Students Prepare Understand Practice Study

Helping Students Prepare Understand Practice Study

Helping Students Prepare Understand Study Yesterday I told you about the interactive test content outline. That was new frontier for us. But we weren’t done being bold. We knew folks were worried about the PSBB section, and the burden that 2 extra courses, one in psychology and one in sociology, would place, disproportionately on the disadvantaged. And MR5’s research caused them to conclude it really wasn’t necessary to add two courses; students could learn this introductory content in a number of different ways. So AAMC reached out to test publishers of psychology and sociology textbooks. We found partners willing to cross reference their textbooks against the PSBB blueprint. Any student can go online, download this list, and find the chapter in a textbook where they can learn about a specific topic. In this case, knowledge is power. And for those of you worried about the cost of another textbook, some of these resources are free, open access. Finally, later this year one of the publishers is coming out with a textbook that integrates the foundational concepts from psychology, sociology, and biology that are important building blocks for the medical school curriculum.

Khan Academy tutorials teach concepts targeted by the new test

What are your colleagues doing? Jorge Girotti Ngozi Anachebe Jorge and Ngozi are going to share efforts at their institutions to help pre-health students get ready for the new exam (each has 5 minutes to share their stories) 48

Activity 2.2 Activity 2.2: Table discussion Participants will have a discussion about solutions they’ve or their colleagues have developed at their institutions to help pre-health students prepare for the new exam. Take some time at your table to discuss the ways that you and your colleagues are helping students get ready for the new exam.