Assessing the Impact of Undergraduate Research and Mentoring (URM) on Student Learning in Biological Sciences Howard R. Mzumara, Ph.D. Director of Testing Center Indiana University – Purdue University Indianapolis (IUPUI) Testing Center Phone: (317) 278-2214 Email: hmzumara@iupui.edu Web site: http://tc.iupui.edu

Presentation Outline Brief Overview of the URM Program in Biological Sciences Basic Information about the URM Program at IUPUI IUPUI URM Project Objectives and Focus of the Evaluation Study Key Evaluation Questions Methodology for the Evaluation Study Mixed-Method Evaluation Approach Qualitative & Quantitative Data Sources Direct & Indirect Measures Issues & Framework for Data Analysis Highlights of Preliminary Study Findings … Evaluation 2011 Conference

Synopsis of URM Program Goal of the URM Program: Increase the number and diversity of individuals pursuing graduate studies in all areas of biological research supported by the National Science Foundation (NSF). Academic Institutions receive support to establish innovative programs to engage undergraduates in year-round research and mentoring activities. URM projects must have strong research and mentoring activities… Particular emphasis is placed on broadening participation of groups historically underrepresented in science and engineering. Evaluation 2011 Conference

IUPUI URM Program Key Strategy: Target pool: Goal: Engage student participants in a multi-year immersion in interdisciplinary research around the theme of biological signaling Target pool: Underrepresented minorities in Biological Sciences (initially recruit 8 URM Fellows per grant year) Goal: Increase the number of underrepresented minorities achieving graduate degrees in Biological Sciences Evaluation 2011 Conference

IUPUI URM Program (continued) Key Program Elements include: Recruitment of URM Fellows & Mentors Faculty Orientation Summer Camp & 9-Week Summer Research Program that include: Multi-tiered Mentoring Sessions & Brown-bag lunches URM-focused Research Sessions / Workshop Series Career Development Sessions Graduate School Boot Camp Poster Session Academic year activities that key into science: URM Fellows spend an average of 10 hrs/wk in labs (to earn at least one research credit per semester); and Attend bi-weekly research cohort meetings Evaluation 2011 Conference

IUPUI URM Project Objectives and Focus of Evaluation Study Key Objective: Assess the effectiveness and impact of the IUPUI URM Program Formative Evaluation Improve design and delivery of the interventions Track effectiveness of project development and implementation Summative Evaluation Assess impact on learners (URM Fellows) Measure knowledge gained by participating in an intensive summer camp/research program Assure project objectives are achieved Evaluation 2011 Conference

Key Evaluation Questions Assessing Effectiveness of the IUPUI URM Program: To what extent did the URM Summer Camp/Research Program enhance student learning or career development? To what extent did the URM Summer Camp/Research Program yield substantive changes in knowledge, attitudes, skills or performance of participants in Biological Sciences? Did the URM Fellows develop a working knowledge of Biological Signaling to the extent that they could design and conduct appropriate research projects? Assessing Impact on internal & external stakeholders: To what degree did the IUPUI URM Program achieve its objectives? In what ways did the URM Summer Camp/Research Program produce benefits among the study participants? Evaluation 2011 Conference

(Mixed-Method Evaluation Approach) Types of Evaluation Measures Direct Measures: Assessments of specific PULs Lab or Course-embedded assessments Assessment of Learning Gains Scores on standardized tests Student products (research papers, Portfolio artifacts) Performance Assessment (Poster/Research Presentations Project documents (e.g., RFP document, progress reports) Indirect Measures: Pre/Post Knowledge or Attitude Surveys (Questionnaires) Research Self-Efficacy Survey Interviews Focus Groups with URM Fellows Participant Evaluations of Workshops Mentor Survey Course/Institutional Data (e.g., enrollment, participation, completion/retention, course grades, and other related data)

Evaluation Instruments/Data Sources Pre/Post Program Surveys Student Attitude/Confidence Scales URM Summer Camp/Research Program Evaluation Surveys Research Self-Efficacy Scale Workshop Evaluations Interviews / Focus Groups with URM Fellows Mentor Evaluation Event Tracking (collect enrollment, participation, and completion data) Extant Data (course grades, GPAs, %DFW, retention, persistence, & graduation rates) extracted from institutional databases Evaluation 2011 Conference

Issues to Address in Data Analysis Small sample size (n) for treatment group Self-selection or research participant bias Lack of an appropriate comparison group Equivalence of Treatment vs. Comparison Groups Appropriateness of theoretical framework Statistical assumptions for using parametric or non- parametric statistical methods to assess project impact or effectiveness of interventions Inherent limitations of self-reported measures Evaluation 2011 Conference

Project Impact Analysis Defining clear objectives for URM Summer Program is essential Assessing clearly articulated outcomes is useful Confidence-building among URM Fellows Skill development, including communication, critical thinking, quantitative reasoning, etc. Career choices affected (e.g., interest in preparing for graduate study or internships in STEM fields) Evaluation 2011 Conference

Project Impact Analysis (continued) Investigate performance and retention of URM Fellows (vs. an appropriate comparison group) Assess impact on student retention and graduation Increase in % of students completing degrees in STEM fields (Compare URM Fellows with an appropriate comparison group that did not participate in summer research activities) Track student self-reported attitudes Where possible, examine the effect on student retention and performance of the project overall and for each of the project strategies Evaluation 2011 Conference

Participant Demographic Summary for Initial Cohort (N=4) Gender Females: 50% Males: 50% Age 18-23 years: 75% 24-29 years: 25% Race/Ethnicity Black/African American: 50% Other Hispanic or Latino: 25% Multiracial: 25% Educational Classification: Junior: 75% Senior: 25% STEM Field Enrolled: Biology: 50% Chemistry: 25% Physics: 25% URM Fellows (Summer 2011) URM Fellows (Summer 2011) Evaluation 2011 Conference

Preliminary Findings … Key Benefits of URM Program (most enjoyable aspects as reported by URM Fellows at IUPUI): One-on-one interaction with different scientists (“mentorship”) Interdisciplinary nature of the URM Program enhanced the understanding of multiple perspectives of Biological Signaling as a research area Sessions on how to prepare for graduate school Scholarships for URM Fellows Evaluation 2011 Conference

Preliminary Findings … (continued) Obstacles encountered/Least enjoyable aspects: Time spent in weekly lab sessions is not helpful Communication of program requirements (e.g., expectations regarding the amount of student work hours per week) is not articulated clearly for URM Fellows, Faculty or Mentors Lack of a glossary of common terms (for use by URM Faculty, Mentors, and Fellows) Evaluation 2011 Conference

Greatest Gains (reported elsewhere)* Readiness for more demanding research Learning a topic in depth Opportunities for presentations Laboratory techniques Enhancement of academic credentials Developing a relationship with a mentor Skill development Learning to persevere Tolerance for obstacles Understanding current research *Source: Adopted from a presentation by Brakke & Crowe (retrieved 11/2/2011 from: http://www.aacu.org/meetings/undergraduate_research/documents/crowebrakke.ppt ) Evaluation 2011 Conference