Assessment of Proposed Research: National Cancer Institute Provocative Questions Initiative Case Study Elizabeth Hsu American Evaluation Association 2012 Annual Meeting October 27, 2012 Co-Authors: Samantha Finstad, Emily Greenspan, Jerry Lee, James Corrigan (NIH) Leo DiJoesph, Duane Williams, Joshua Schnell (Thomson Reuters)

Outline Analysis Questions Applicant characterization study Novelty and relevance study

Outline Analysis Questions Applicant characterization study Novelty and relevance study

Analysis Questions What was the breadth of the investigators who applied to the PQ initiative? Did the PQ initiative compel investigators to propose new lines of research? -“Novelty” as compared to self and general Did the ideas presented in the applications fill gaps in the NCI portfolio, as identified by the PQ RFA? -“Relevance” as compared to RFA text

Outline Analysis Questions Applicant characterization study Novelty and relevance study

Methodology: Applicant Characterization Discipline -Determined using combination of degree, institution department, primary degree field, expertise, and PI biosketch -MD and MD/PhD not further characterized Investigator status -NIH definition of new investigator and early stage investigator -Flags in NIH database; suspect flags confirmed using prior grant history and date of degree information Multiple PI applications

Applicant Characterization Investigator Status% of PQ Applicants New Investigator20.7% Early Stage Investigator (NI Subset)15.1% Experienced Investigator64.2% Investigator Discipline% of PQ Applicants Basic/life sciences47.7% Behavioral1.6% Epidemiology2.1% Physical science/engineering13.3% MD15.3% MD/PhD20.0% Multi-PI Applications% of PQ Applications 1 PI77.7% 2 PI17.5% 3+ PI4.8% NIH awarded PIs: ~30% MD or MD/PhD NCI R01/R21 applicants: ~30% MD or MD/PhD NIH R01: ~30% New Investigator/Early Stage Investigator NCI R01: ~33% New Investigator/Early Stage Investigator

Applicant Characterization PQ RFA attracted: -More new investigators/early stage investigators -More MD and MD/PhDs Every PQ received multi-PI applications

Outline Analysis Questions Applicant characterization study Novelty and relevance study Methodology Results Conclusions

Methodology: Novelty and Relevance Text similarity algorithm, as previously described Self novelty Compare PQ application to prior applications by same PI General novelty Compare PQ application to sample of NIH applications, excluding PI’s own applications Relevance Compare PQ application to question text (question, background, feasibility, implications of success) as posed in RFA Scaled from 0 to 1 Thresholds chosen by manual review

Methodology: Measurement Goals 754 PQ applications received FY ,088 comparison cohort NIH applications FY ,008 HHS/NIH applications found as similar to the PQ RFA text in the portfolio analysis study FY Provocative Question (PQ) RFAs FY 2012 Pre-RFA Portfolio Analysis “Coincidental” Relevance Relevance Novelty

Methodology: Plotting Text Similarity Scores 01 Text Distance Decreasing Document Similarity Increasing Document Similarity Not RelevantRelevant Not Novel Novel Distance (Comparing RFA text to PQ Application text) Distance (Comparing PQ Application text to Prior Application text)

Outline Analysis Questions Applicant characterization study Novelty and relevance study Methodology Results Did the PQ initiative compel investigators to propose new lines of research? Did the ideas presented in the applications fill gaps in the NCI portfolio, as identified by the PQ RFA? Conclusions

Did Applicants Change Their Research Direction? In general, applicants stayed close to their prior lines of research Median novelty distance by- self = 0.08 Notable exceptions where investigators appeared to move far from prior applications Multiple outliers with high novelty scores found in questions 5,11,17,18,21,23

Were Applications Similar to Other Existing NIH Research (non-self)? Questions with largest proportion of novel applications relative to the NIH-wide set were: - 3 (100%) - 2 (87%) -13 (81%) -12 (75%) - 4 (73%) Comparison group is needed to better understand the general trends.

Did the PQs Compel Investigators to Propose New Lines of Research? PQ applicants generally did not stray far from previous lines of research Subtle scientific differences may result in low similarity scores PQ application text was generally more similar to the PI’s previous work than to NIH-wide pool Comparison group is needed to evaluate how the similarity seen here compares to responses to other RFAs

Outline Analysis Questions Applicant characterization study Novelty and relevance study Methodology Results Did the PQ initiative compel investigators to propose new lines of research? Did the ideas presented in the applications fill gaps in the NCI portfolio, as identified by the PQ RFA? Conclusions

Did Applications Address Gaps in Scientific Research as Intended by the RFA? Relative Ranking of Relevance: PQ vs. Portfolio Analysis Applications N = , PQ RFA ResponsePortfolio Analysis Application Distance Question Number

Did Applications Address Gaps in Scientific Research as Intended by the RFA? Proxy examination question: Did the text of the PQ applications show a closer relationship to the RFA text compared to previously existing grants? -Drawbacks of using RFA text May refer to specific examples (e.g., particular cancer site or drug), biasing towards applications that refer to that example Some applications repeat the RFA text verbatim Application Level -613 of applications were found to be relevant. Question Level -Some questions received only a small percentage of applications below threshold -Some questions received applications that were generally no more responsive than previously existing applications -Questions that received the most responsive applications were 2, 16, 22 and 24

Were Applications Generally Novel and Relevant? Yes No NI or ESI Relevance = 1 – rel. distance Minimum Novelty Distance Relevance and Best (Minimum) By-Self Novelty of PQ Applications with Investigator Status PQ 1 (n = 77) PQ 5 (n = 64)PQ 14 (n = 47)PQ 17 (n = 29) Relevance and Best (Minimum) General Novelty of PQ Applications with Investigator Status PQ 1 (n = 84)PQ 5 (n = 67)PQ 14 (n = 50)PQ 17 (n = 32)

Outline Analysis Questions Applicant characterization study Novelty and relevance study Methodology Results Conclusions

Methodology Conclusions Novelty score detected applications that were very similar to prior work with high recall Challenging to distinguish scientific nuances and extensions of previous work Current relevance measurement is dependent upon the target text Comparison group may provide a more thorough understanding of both the novelty and relevance metrics Lessons learned -A larger cohort is needed to avoid missing novelty spoilers -Similarity measurements are sensitive to many factors, including cohort size and scaling parameters -Specific aims may be more informative than abstract text for measuring relevance

Questions?

Applicant Discipline Basic/Life sciences: 47.7% Behavioral: 1.6% Epidemiology: 2.1% MD: 15.3% MD/PhD: 20.0% Physical science/Engineering: 13.3%

Applicant Investigator Status New Investigator: 20.7% Early Stage Investigator: 15.1% Experienced Investigator: 64.2%

Multi-PI Applications 1 PI: 77.7% 2 PI: 17.5% 3+ PI: 4.8%

Did Applications Address Gaps in Scientific Research as Intended by the RFA? , , , , , ,155 Relative Ranking of Relevance: PQ Applications vs. Portfolio Analysis Applications PQ RFA ResponsePortfolio Analysis Application n (PQ) = n (Portfolio Analysis) =

Were Applications Generally Novel and Relevant? The majority of questions were found to be similar to the applicant’s prior work and relevant to the RFA