1. University of Washington, Seattle Buffalo, NY October, 2017
Review of Phase I Pilot and Recommendations for Next Step in Scaling Up EvaluateUR
University of Washington, Seattle
Buffalo, NY
October, 2017

2. Institution Profile University of Washington – R1 Research University
The UW is a multi-campus university in Seattle, Tacoma and Bothell, as well as a world-class academic medical center.
Enrollment – Seattle Campus
Undergraduate: 31,418
Graduate/Professional: 14,663
TOTAL: 46,081
1.37 billion annually in grants and contracts for research
Undergraduate research part of the culture and fabric of the institution
Over 8K undergraduate researchers worked with faculty mentors primarily during academic year ( )
Over 1200 undergraduates presented their work in the 20th Annual UW Undergraduate Research Symposium on May 19, 2017

3. Summer Research Program
Since 2002, directly administer and co-host intensive summer research programs in various academic areas; student participants/summer
Components: weekly seminar, social and prof. development activities, advising, poster session
Collaborate with other campus programs: Welcome BBQ and Summer Research Poster Session; students
UW Summer 2017 STEM Programs
# of students
Center for C-H Functionalization (CCHF) 1
Clean Energy Bridge to Research REU 12
CoMotion Mary Gates Innovation Scholars Summer Program 9
HHMI Exceptional Research Opportunities Program 3
Institute for Protein Design (IPD) Undergraduate Fellows 2
Rosetta Commons REU Program
Scan Design Innovations in Pain Research Summer Program
TOTAL 37

4. Summer Research Program (cont.)
Chose to pilot EvaluateUR with Scan Design Innovations in Pain Summer Research Program
Good rapport w/ PI in Anesthesiology who is interested in evaluation
All 9 student-mentor pairs participated
Started in 2015; exposes undergraduate students to interdisciplinary research in pain to stimulate interest in future careers in pain research
Immersive summer research experiences in both basic science and clinical research in the treatment of chronic pain – MD, PhD, and MD/PhD aspirations

5. UR Program Director Profile – Jennifer Harris, PhD
M.Ed. In Educational Leadership & Policy Studies, UW PhD in Social and Cultural Foundations of Education, UW
Full-time administrator - director of Undergraduate Research Program since 2014; started as assistant director in 2005
Responsible for creating the overall structure and content of the program, recruiting faculty mentors, orienting students and mentors, and supporting the participants throughout the program
Work closely with associate vice provost for research and other program staff assist with organizing activities, advising students, and interacting with mentors:
Janice DeCosmo, PhD, Associate Vice Provost for Research and Associate Dean, Undergraduate Academic Affairs
Jessica Salvador, PhD, Assistant Director, URP
Dan Herb, M.Ed., Adviser and Program Coordinator, URP

6. Phase I Participation Overview
9 student-mentor teams
6 students participated with faculty mentor
2 participated with research staff mentors and 1 with a clinical resident
Dashboard step modifications
Mid-Research
Replaced “faculty mentor” language with “mentor” to reflect different positions of mentors
Changed STEP 7: Rather than notifying us when reach research midpoint, set midpoint date – result is we removed STEP 8 and had 16 rather than 17 steps.
End-of-Research
Removed reference to stipend payments; upload of abstract and poster
Final Research Report changed to Program Feedback; retained questions and added UW program-specific feedback items

7. Progress Completing Dashboard Steps

8. Orientation Conducted three 45-minute EvaluateUR Orientation Sessions
Zoom conferencing with Jill Singer for each
Handouts included
Buffalo State Slides
EvaluateUR Outcome Categories (from CUR article)
UW EvaluateUR Timeline & Activities – adapted from Buffalo State

9. UW EvaluateUR Timeline & Activities
Distributed at Orientation

10. Tracking
Created a visual schedule with target dates that was shared with participants
Watched the dashboard to track progress
nudging: Sent frequent reminders to participants
In final phase, also phoned a couple of students
Despite these efforts, 2 of the 9 pairs did not complete

11. What Worked Well
User interface is easy to use and we were able to customize to meet our program needs
Orientation sessions provided context for and excitement around EvaluateUR
Students
Appreciated structured appointment times for conversation, reflection, and feedback
Noted tool helped them to think about their development as researchers

12. Your Top 3 Challenges
Keeping participants on task – nudging takes time and energy
Mentors and students utilize EvaluateUR tools not necessarily in the ways intended
E.g., heard from mentors and mentees that discussion meetings did not focus on the research assessments but how things going generally
Several mentors wanted more flexibility/input to edit content
Mentors liked general outcome categories but found it challenging to rate all of the aspects – noted students just starting and only know them for short period of time – and didn’t want to give low scores. Wanted “not observed” option
Suggestion made to offer a question library around general outcome category concepts and they could be given options to choose pre- summer
This is #4 I know!: How to work with collaborative mentoring teams

13. Your Top 3 Lessons Learned
Timing/efficiency is critical
Orientation – consider doing EvaluateUR Orientation on Day One of the program
Consider streamlining the Pre-Research Assessment Steps
Students fill out the Pre-Research Reflection prior to orientation > mentor takes a few minutes to review during orientation > mentor and student break out and have their conversation regarding the Pre-Research Reflection during the session
Provide additional assistance in orientation
How to respond to survey items they may find challenging
Best practices for utilizing assessments in mentor-mentee meetings
Unforeseen circumstances occur
Sometimes babies arrive early!

14. Lessons Learned and Advice
Get mentors and mentees together for orientation, streamline pre-research assessment steps, calendar meetings
Manage mentor expectations and perspectives
Introduce them to the tool early
Be prepared for mentor feedback on the tool
Start IRB early – our campus requested clarifications and additional information