1. NIH LOAN REPAYMENT PROGRAM EVALUATION Milton J. Hernández, Ph.D. Director Division of Loan Repayment OEP, OER, OD National Institutes of Health Bethesda, MD NIH Loan Repayment Programs Extramural LRPs FY 2003-2009 Laurel Haak, Ph.D. Chief Science Officer Discovery Logic, Inc. Rockville, MD

2. 2 NIH LRP Evaluation Questions  Question 1: Who is applying to the NIH LRP?  Question 2: Does participation in the LRP affect career outcomes?  Question 3: How does the LRP compare to other NIH career development award programs? NIH Loan Repayment Programs

3. 3 New and Renewal LRP Applications and Awards Trends Analysis, FY03-FY07 NEW RENEWAL NIH Loan Repayment Programs The renewal program has decreased the success rate for new applications but has not affected the number of new applications submitted. The success rate for new applications from FY05-FY07 is 40% and for renewals is 70%.

4. 4 Gender Distribution of New LRP Applicants Trends Analysis, FY03-FY07 NIH Loan Repayment Programs  The ratio of women to men LRP applicants is reversed from the ratio in recent graduating MD and PhD classes.  Success rates are not different.

5. 5 Gender Distribution of New LRP Applicants, by Degree Trends Analysis, FY03-FY07 NIH Loan Repayment Programs Degree TypeExample Degrees Physician DoctorateMD, DO Academic DoctoratePhD, ScD, EdD, DPH Physician Doctorate / Academic Doctorate MD/PhD, DO/ScD Professional Clinical Doctorate DDS, DMD, DVM, PharmD, PsyD Allied Health Professional BSN, MPH, MSN About the same number of men and women applicants are MDs, but 2.5 times more women applicants are PhDs, and twice as many men applicants hold dual degrees. p<0.001

6. 6 Race/Ethnicity of New LRP Applicants Trends Analysis, FY03-FY07 NIH Loan Repayment Programs  More Blacks and fewer Asians applied as compared to recent MD and PhD graduating classes.  Success rates did not differ from each other or from the overall success rate.

7. 7Prepared by NIH OER/OEP/DLR Educational Debt, by Degree Type Trends Analysis, FY03-FY07 Among LRP applicants, academic doctorates have a higher debt load than recent PhD graduates, and physician doctorates have a lower debt load than recent MD graduates. †Source: Survey of Earned Doctorates, National Science Foundation (2003). * Source: Survey of Graduates, American Association of Medical Colleges (2003-2007).

8. 8 Years Since Degree Trends Analysis, FY03-FY07 NIH Loan Repayment Programs LRP attracts early career researchers  72% of new applicants apply within 6 years of receiving their degree  90% of new applicants apply within 9 years of receiving their degree Academic doctorates and physician doctorates apply at different times  67% of academic doctorates apply within 3 years of their degree  68% of physician doctorates apply 4-6 years after receiving their degree 4-6 Years (37%) Up to 3 Years (35%) 7-9 Years (18%) 10+ Years (10%)

9. 9 NIH LRP Evaluation: Outcome Analysis  Question 1: Who is applying to the NIH LRP?  Question 2: Does participation in the LRP affect career outcomes?  Question 3: How does the LRP compare to other NIH career development award programs? NIH Loan Repayment Programs

10. 10 Outcome Analysis Methodology  Outcome analysis: Regression Discontinuity Analysis.  We compared two nearly-identical applicant groups. LRP applicants with very similar priority scores were identified, about half of whom were funded and half of whom were not funded. NIH Loan Repayment Programs Not Funded Funded Priority Score “Bubble”

11. Identifying the Comparison Cohort 11 NIH Loan Repayment Programs F8F8 NF 8 NIAID, FY05 F9F9 NF 9 NIDDK, FY04 F 11 NF 14 NICHD, FY04 Identify bubble applicants for each IC and each fiscal year F 259 NF 254 IC-FY Bubble 513 Combine individual bubbles Priority Score range, e.g. 190-214 F8F8 NF 9 NIMH, FY03 Funded Not Funded A 50-50 mix of funded and not funded applicants with closely-matched priority scores Outcome Analysis Subsequent publications and NIH grant participation LRP Applicant Cohort, FY03-FY06

12. IC-FY Bubble, Demographic Distribution ParameterCategory % Funded IC-FY Bubble (n=259) % Not Funded IC-FY Bubble (n=254) % IC-FY Bubble (n=513) % All New Applicants, FY03-FY07 (n=9820) Gender Male41%36%38%44% Female57%62%60%53% Degree Type Physician Doctorate48%46%47% Professional Clinical Doctorate 2%6%4%5% Academic Doctorate44%45% 42% Physician Doctorate / Academic Doctorate 5%2%3%6% Allied Health Professional1% <1% Years Since Degree Up to 3 Years 52%57%54% 35% 4-6 Years 34%35% 37% 7-10 Years 11%6%8% 18% 10+ Years 2%3% 10% 12 Outcome Analysis, LRP Applicant Cohort, FY03-FY06 The demographics of the funded and not funded IC-FY Bubble cohorts are indistinguishable, hence they can be used as comparison groups. There are some notable differences compared to the total new applicant group.

13. 13 p=0.0102 NIH Grant Participation: IC-FY Bubble Overall

14. NIH Grant Participation: IC-FY Bubble, by Gender Outcome Analysis, LRP Applicant Cohort, FY03-FY06 MaleFemale 14NIH Loan Repayment Programs Women LRP awardees are less likely than men to have applied for an NIH award or to have a subsequent observable role in the NIH extramural program. p=0.0474 p=0.0147

15. NIH Grant Participation: IC-FY Bubble, by Degree Outcome Analysis, LRP Applicant Cohort, FY03-FY06 Physician DoctorateAcademic Doctorate 15NIH Loan Repayment Programs Participation in the LRP program increased the likelihood that a physician doctorate applied for a subsequent NIH award and was retained in the NIH extramural workforce. It also increased the likelihood that an academic doctorate was awarded an NIH grant. p=0.0106 p=0.0470 p=0.0324

16. Question 3: Comparison with Other NIH Programs 16NIH Loan Repayment Programs LRP Cohort Outcome analysis Subsequent NIH grant participation Research accomplishments Publications Demographic parameters Identification of Comparison Cohorts LRP Top-10% n = 215 K Top-10% n = 218 K Cohort K01, K08, K23 LRP+K Cohort LRP+K Top-10% n = 96 Funded Type I awardees, FY03-FY06 Top 10% priority score, each FY/IC L awards only Funded Type I awardees, FY03-FY06 Top 10% priority score, each FY/IC K awards only Funded Type I awardees, FY03-FY06 Top 10% priority score, each FY/IC L and K awards* *L award between FY03-FY06 and a K01/K08/K23 award at any time. Awardees were in the top 10% of priority scores for both award mechanisms.

17. NIH Grant Participation: K and L Programs Outcome Analysis, 0-6 Years Since Degree, Top-10% Comparison 17NIH Loan Repayment Programs Even after balancing for years since degree, the LRP recipients show less likelihood of applying for subsequent NIH funding than K or LRP+K awardees. LRP awardees are also less likely than K awardees to have a subsequent role on NIH awards. p=0.0003 p=0.0176 p=0.0003