1. A National Evaluation of the RISE Program Wesley Schultz, Ph.D. Anna Woodcock Department of Psychology California State University, San Marcos Presented at the 2006 NIH Grantee Meeting, “Efficacy of Interventions to Promote Research Careers.” August 21, Washington, DC.

2. Success, By What Comparison? Black, Hispanic, and Native Americans are underrepresented at all levels of education 27% of population 16% of BS 12% of MS 9% of Ph.D. 8% of University Faculty (unknown by science) Trends show substantial increase (nearly doubling) in past 20 years. 80% increase in minority Ph.Ds since 1977 62% increase in minority population during same period

3. Source: Compiled Sources from Higher Education Research Institute (HERI) Historical Trends in Higher Education for Underrepresented Students

4. Measuring Success 1. National-level Data 80% increase in minority Ph.Ds since 1977 62% increase in minority population during same period 2. Site-specific Data Each NIH program submits yearly progress reports, and longer-term evaluation results Typically track their student success, but what about a control group?

5. Measuring Success 3. Large-Scale Evaluation Assessment of NIH Minority Research and Training Programs (Phase III). Science, 2006. Surveyed students from 47 programs (1970 -99) Substantial limitations in finding former students 5,371 identified participants, of which data was obtained from 739 (only 83 trainees funded as undergraduates-- 19 U*STAR, 31 Bridges, 33 COR) “The outcomes for the MARC program are less than anticipated, with less than 15% of all MARC students completing the Ph.D. in biomedical or behavioral sciences” (NIGMS White Paper, 2006)

6. Problems with Existing Data 1. Programs can “cherry pick” students who are likely to succeed - These students are likely to succeed without the program (growing new talent versus harvesting abilities) 2. No control group 3. Long-term evaluations outside of funding scope 4. No way to examine the “mechanisms” of success 5. Students lose contact after leaving the program - Will the recommended 2008 MDS (NIH Minimum Data Set) help?

7. TheScienceStudy.com Longitudinal study of RISE and MARC students Participants from 25 programs nationwide Yearly (or twice-yearly) surveys from students Matched control For each RISE or MARC student, we find a similar student who does not go through the program Matching variables: ethnicity, gender, major, GPA, intention to become a scientist, enrollment level (LD, UD, Grad) Secondary matching: age, parental education, community college transfer, English as first language

9. Longitudinal Study Anticipated final sample 450 RISE/MARC 450 Matched Control All data collected through web interface www.TheScienceStudy.com Survey measures psychological constructs, student accomplishments, aspects of the program

11. Year 1 Accomplishments 1. Recruited panel of RISE and MARC students 2. Recruited matched panel members 3. Surveyed RISE directors 4. Wave 1 survey 45 minute on-line and CATI survey Measures: demographics, accomplishments, aspirations, training, mentoring, financial support, identity, esteem, efficacy, learning goals

12. Recruitment of RISE and MARC students Panel size: 870 273 current RISE students, 65 MARC, 97 other- funded Screened 1,684 potential match students and selected 435 Explored various matching procedures (e.g., propensity scores) Opted for an equal-weighting procedure of 10 variables

13. Survey of RISE Directors 24 completed surveys (100%) One program stopped operating RISE Directors provided aggregated demographic data on their students Overall capture rate: 57% (Cliff could help here) No significant differences between reported demographics and students in our panel (gender, age, ethnicity, ESL, major, educational status)

14. Panel Representativeness Percentage of RISE Students by Ethnicity in Collaborating Programs and on the Science Study Panel Ethnicity RISE Program % Science Study Panel % Asian1%5% African American59% Hispanic/Latino(a)30%29% Native American2%3% Hawaiian2%1% White4%3% Other2%0% Total100%

15. Survey of RISE Directors 34% Gender breakdown of RISE students in collaborating programs and on the Science Study panel

16. Survey of RISE Directors RISE Program Graduates and Matriculants to Graduate School Total% How many of your RISE students are graduating this year (May, 2006)? 96 How many are going into a Ph.D. program?3334% How many are going into a Masters program?2223% Total # to graduate school5557%

17. Survey of RISE Directors RISE Program Graduates and Matriculants to Graduate School Total% How many of your RISE students are graduating this year (May, 2006)? 96 How many are going into a Ph.D. program?3134% How many are going into a Masters program?2223% Total # to graduate school5557%

18. Survey of RISE Directors Activities and support offered by collaborating RISE programs during 2005/06.

19. Survey of RISE Directors Activities rated by RISE Directors as the most beneficial to students.

20. Wave 1 Surveyed 870 panel members Mixed-mode survey (online and CATI) Eight week survey protocol 85% response rate

21. Wave 1 Panel

22. Year 2 Second round of panel recruitment (RISE, MARC, match) Two waves of surveys (Fall, Spring) RISE Director survey Online data portal (project home to facilitate longitudinal contacts) Full panel, with three data points

24. ScaleSource # of Questions Faculty Mentoring Relationship - Psychosocial Support & Networking Dreher and Ash (1990). 15 Faculty Mentoring Relationship - Interpersonal ComfortOrtiz-Waters & Gilson (2005)2 Faculty Mentoring Relationship - Perceived SimilarityTurban and Jones (1988)2 Faculty Mentoring Relationship - Mentor SatisfactionEnsher & Murphy (1997)3 Science Identity (1)Serpe (2006)4 Science Identity (2)UCSC Science Undergraduate Survey. Martin Chemers.14 Self-EsteemRosenberg Self-Esteem Scale10 Academic IdentitySerpe (2006)5 Task Goal Orientation Midgley, C., Kaplan, A., Middleton, M., Maehr, M., Udran, T., Anderman, L. H., Anderman, E., & Roeser, R. (1998) 6 Ability-Approach OrientationMidgley et al. (1998)6 Ability-Avoid OrientationMidgley et al. (1998)6 Study Strategies: Deep ProcessingElliot, McGregor, & Gable (1999)5 Study Strategies: PersistenceElliot, McGregor, & Gable (1999)4 Study Strategies: EffortElliot, McGregor, & Gable (1999)2 Confidence as a ScientistUCSC Science Undergraduate Survey. Martin Chemers.14 Confidence in Leading and Working on a Research TeamUCSC Science Undergraduate Survey. Martin Chemers.12 Stereotype ThreatSteele, James, & Chait Barnett (2002)8 The Multigroup Ethnic Identity Measure (MEIM)Phinney, J. (1992).12 Scales

26. Questions We Can Answer 1. Does participating in the RISE/MARC program cause an increase in the likelihood that a minority student will pursue a career in the biomedical sciences? 2. What are the types of activities that students involved in RISE/MARC programs are exposed to? 3. Are there some types of students who benefit more from the RISE/MARC program than others? 4. Are there elements of the RISE/MARC program that are linked with the success of the students? (Research experience, Faculty mentoring, Motivated peers, Resume of accomplishments)