University of Washington, Seattle Buffalo, NY October, 2017 Review of Phase I Pilot and Recommendations for Next Step in Scaling Up EvaluateUR University of Washington, Seattle Buffalo, NY October, 2017

Institution Profile University of Washington – R1 Research University The UW is a multi-campus university in Seattle, Tacoma and Bothell, as well as a world-class academic medical center. Enrollment – Seattle Campus Undergraduate: 31,418 Graduate/Professional: 14,663 TOTAL: 46,081 1.37 billion annually in grants and contracts for research Undergraduate research part of the culture and fabric of the institution Over 8K undergraduate researchers worked with faculty mentors primarily during academic year (2015-16) Over 1200 undergraduates presented their work in the 20th Annual UW Undergraduate Research Symposium on May 19, 2017

Summer Research Program Since 2002, directly administer and co-host intensive summer research programs in various academic areas; 20-60 student participants/summer Components: weekly seminar, social and prof. development activities, advising, poster session Collaborate with other campus programs: Welcome BBQ and Summer Research Poster Session; 100-150 students UW Summer 2017 STEM Programs # of students Center for C-H Functionalization (CCHF) 1 Clean Energy Bridge to Research REU 12 CoMotion Mary Gates Innovation Scholars Summer Program 9 HHMI Exceptional Research Opportunities Program 3 Institute for Protein Design (IPD) Undergraduate Fellows 2 Rosetta Commons REU Program Scan Design Innovations in Pain Research Summer Program TOTAL 37

Summer Research Program (cont.) Chose to pilot EvaluateUR with Scan Design Innovations in Pain Summer Research Program Good rapport w/ PI in Anesthesiology who is interested in evaluation All 9 student-mentor pairs participated Started in 2015; exposes undergraduate students to interdisciplinary research in pain to stimulate interest in future careers in pain research Immersive summer research experiences in both basic science and clinical research in the treatment of chronic pain – MD, PhD, and MD/PhD aspirations

UR Program Director Profile – Jennifer Harris, PhD M.Ed. In Educational Leadership & Policy Studies, UW PhD in Social and Cultural Foundations of Education, UW Full-time administrator - director of Undergraduate Research Program since 2014; started as assistant director in 2005 Responsible for creating the overall structure and content of the program, recruiting faculty mentors, orienting students and mentors, and supporting the participants throughout the program Work closely with associate vice provost for research and other program staff assist with organizing activities, advising students, and interacting with mentors: Janice DeCosmo, PhD, Associate Vice Provost for Research and Associate Dean, Undergraduate Academic Affairs Jessica Salvador, PhD, Assistant Director, URP Dan Herb, M.Ed., Adviser and Program Coordinator, URP

Phase I Participation Overview 9 student-mentor teams 6 students participated with faculty mentor 2 participated with research staff mentors and 1 with a clinical resident Dashboard step modifications Mid-Research Replaced “faculty mentor” language with “mentor” to reflect different positions of mentors Changed STEP 7: Rather than notifying us when reach research midpoint, set midpoint date – result is we removed STEP 8 and had 16 rather than 17 steps. End-of-Research Removed reference to stipend payments; upload of abstract and poster Final Research Report changed to Program Feedback; retained questions and added UW program-specific feedback items

Progress Completing Dashboard Steps

Orientation Conducted three 45-minute EvaluateUR Orientation Sessions Zoom conferencing with Jill Singer for each Handouts included Buffalo State Slides EvaluateUR Outcome Categories (from CUR article) UW EvaluateUR Timeline & Activities – adapted from Buffalo State

UW EvaluateUR Timeline & Activities Distributed at Orientation

Tracking Created a visual schedule with target dates that was shared with participants Watched the dashboard to track progress Email nudging: Sent frequent email reminders to participants In final phase, also phoned a couple of students Despite these efforts, 2 of the 9 pairs did not complete

What Worked Well User interface is easy to use and we were able to customize to meet our program needs Orientation sessions provided context for and excitement around EvaluateUR Students Appreciated structured appointment times for conversation, reflection, and feedback Noted tool helped them to think about their development as researchers

Your Top 3 Challenges Keeping participants on task – email nudging takes time and energy Mentors and students utilize EvaluateUR tools not necessarily in the ways intended E.g., heard from mentors and mentees that discussion meetings did not focus on the research assessments but how things going generally Several mentors wanted more flexibility/input to edit content Mentors liked general outcome categories but found it challenging to rate all of the aspects – noted students just starting and only know them for short period of time – and didn’t want to give low scores. Wanted “not observed” option Suggestion made to offer a question library around general outcome category concepts and they could be given options to choose pre- summer This is #4 I know!: How to work with collaborative mentoring teams

Your Top 3 Lessons Learned Timing/efficiency is critical Orientation – consider doing EvaluateUR Orientation on Day One of the program Consider streamlining the Pre-Research Assessment Steps Students fill out the Pre-Research Reflection prior to orientation > mentor takes a few minutes to review during orientation > mentor and student break out and have their conversation regarding the Pre-Research Reflection during the session Provide additional assistance in orientation How to respond to survey items they may find challenging Best practices for utilizing assessments in mentor-mentee meetings Unforeseen circumstances occur Sometimes babies arrive early!

Lessons Learned and Advice Get mentors and mentees together for orientation, streamline pre-research assessment steps, calendar meetings Manage mentor expectations and perspectives Introduce them to the tool early Be prepared for mentor feedback on the tool Start IRB early – our campus requested clarifications and additional information