1. Outcome Evaluation of the National Cancer Institute (NCI) Career Development (K) Awards Program Julie L. Mason, Ph.D. Center for Cancer Training National Cancer Institute National Institutes of Health

2. 2 K Program Background 3-5 years support to early-career scientists, clinicians, health professionals Facilitate independent cancer research careers Broad range of K grant mechanisms No systematic evaluation of the NCI K program performed Feasibility study, outcome evaluation

3. Evaluation Objectives Determine demographics of NCI K award program applicants Identify post-K outcomes Determine impact of NCI K award receipt on participant outcomes 3

4. 4 Full Evaluation Cohort Mechanism Years Analyzed Total Applications Total Awards Percent Awarded K011997 - 200762415324.52% K041970 - 19961,24934127.30% K071980 - 200882527433.21% K081984 - 20081,63851531.44% K111987 - 19962168639.81% K121992 - 20081154438.26% K221998 - 20083247322.53% K231999 - 20083669826.78% K252000 - 2008882528.41% TOTAL1970 - 20085,4451,60929.55% Mechanism Applicants Awardees Non- Awardees Percent Awarded TOTAL3,982 1,547 + 188 Scholars 2,43538.85%

5. 5 Outcome Evaluation Framework INPUTSACTIVITIESOUTCOMES AND IMPACT Implement career development plan Perform research Transition to independence Qualifying degree Degree field of study Gender Race/ethnicity Years since degree Age at application Prior NIH support Institutional characteristics K Applicants Goals Mechanisms included Timeframe evaluated Number of K applications and applicants K success rates K Program  NSF Doctorate Records File  AAMC Faculty Roster  IMPAC II  NCI Fact Book

6. Creation of the Comparison Cohort Cutting point (payline) n = 293 FUNDED K awardees NOT FUNDED K non-awardees Least fundable scores Most fundable scores 6

7. 7 Outcome Evaluation Framework ACTIVITIES Implement career development plan Perform research Transition to independence NIH applications & awards NCI applications & awards Non-NIH awards Time to first R01 Subsequent Funding  IMPAC II  International Cancer Research Partnership (ICRP)  ScienceWire ® (DoE, NSF) OUTCOMES AND IMPACT

8. 8 Outcome Evaluation Framework ACTIVITIES Implement career development plan Perform research Transition to independence NIH applications & awards NCI applications & awards Non-NIH awards Time to first R01 Subsequent Funding  IMPAC II  International Cancer Research Partnership (ICRP)  ScienceWire ® (DoE, NSF) OUTCOMES AND IMPACT YES (1)NO (0) YES (1) 711495 NO (0) 6101,077 K Funded Had Subsequent NIH Grant OR = 2.53, p<0.001  IMPAC II  International Cancer Research Partnership (ICRP)  ScienceWire ® (DoE, NSF)

9. Outcome Evaluation Framework 9  IMPAC II  MEDLINE  Web of Science  ClincialTrials.gov  US Patent and Trade Office ACTIVITIES Implement career development plan Perform research Transition to independence OUTCOMES AND IMPACT NIH applications & awards NCI applications & awards Non-NIH awards Time to first R01 Subsequent Funding Number of publications Citations and impact Patents awarded Clinical trial participation Productivity

10. Outcome Evaluation Framework NIH applications & awards NCI applications & awards Non-NIH awards Time to first R01 Subsequent Funding 10 Number of publications Citations and impact Patents awarded Clinical trial participation Productivity  IMPAC II  MEDLINE  Web of Science  ClincialTrials.gov  US Patent and Trade Office ACTIVITIES Implement career development plan Perform research Transition to independence OUTCOMES AND IMPACT

11. 11 Outcome Evaluation Framework Number of publications Citations and impact Patents awarded Clinical trial participation Productivity Career Appointments Faculty appointment Participation on advisory committees & panels Professional society membership Registered health practitioner listing NED listing  IMPAC II  AAMC Faculty Roster  HealthLink/Lodestone  NIH Employee Directory  FIDO.gov  AACR  FASEB  ASCO ACTIVITIES Implement career development plan Perform research Transition to independence NIH applications & awards NCI applications & awards Non-NIH awards Time to first R01 Subsequent Funding OUTCOMES AND IMPACT

12. 12 Outcome Evaluation Framework Faculty appointment Participation on advisory committees & panels Professional society membership Registered health practitioner listing NED listing Career Appointments Funded research activity Broader scientific engagement Number of publications Citations and impact Patents awarded Clinical trial participation Productivity OUTCOMES AND IMPACT NIH applications & awards NCI applications & awards Non-NIH awards Time to first R01 Subsequent Funding

13. 13 Funded Research Activity Awarded a competitive grant from NIH, Dept. of Energy (DoE), or International Cancer Research Partnership (ICRP), or Key personnel on a clinical trial Is a funded researcher: K applicants

14. 14 Funded Research Activity Full cohort Comparison cohort K Awardees n = 1,204 K Non-awardees n = 1,685 K Awardees n = 293 K Non-awardees n = 293 p<0.001 p=0.0076

15. 15 Broader Scientific Engagement Is engaged: Federal advisory committee or review panel Published subsequent papers Matched to NIH Employee Directory (NED) Membership in scientific professional societies Registered health practitioner Is a funded researcher Not a funded researcher Did not include individuals in the “Is a funded researcher” category

16. 16 Broader Scientific Engagement Is engaged: Federal advisory committee or review panel Published subsequent papers Matched to NIH Employee Directory (NED) Membership in scientific professional societies Registered health practitioner Is a funded researcher Not a funded researcher Did not include individuals in the “Is a funded researcher” category

17. 17 Broader Scientific Engagement Full cohort Comparison cohort K Awardees n = 407 K Non-awardees n = 927 K Awardees n = 111 K Non-awardees n = 144 p<0.001 p=0.0012

18. 18 Jonathan Wiest, NCI Ming Lei, NCI Jessica Faupel-Badger, NCI Erika Ginsburg, NCI James Corrigan, NCI Lawrence Solomon, NCI Melissa Stick, NIDCD Joanne Tornow, NSF Rodney Ulane, NIH OD Joshua Schnell Laurel Haak Yvette Seger Leo DiJoseph Faye Liu John Seibel Thomson Reuters/Discovery LogicAdvisory Committee Data Partners NSF Doctorate Records File AAMC Faculty Roster American Association for Cancer Research American Society of Clinical Oncology Federation of American Societies for Experimental Biology International Cancer Research Partners Partial Funding NIH Evaluation Set-Aside Program Acknowledgements

19. 19 Additional Questions? Julie L. Mason, Ph.D. MASONJL@MAIL.NIH.GOV Tel: 301-435-5163 Center for Cancer Training National Cancer Institute http://cancer.gov/cct