American Evaluation Association Annual Meeting, November 2011 Using Multiple Methods and Data Sources to Analyze Complex Cancer Research Portfolios Joshua Schnell Discovery Logic, A Thomson Reuters Business Co-Authors: Elizabeth Hsu, Jim Corrigan (NIH), Sandeep Patel, Lauren Taffe, Duane Williams (Discovery Logic)

American Evaluation Association Annual Meeting, November 2011 Data-driven portfolio management The National Institutes of Health seeks to correlate the research and training it funds with subsequent public health impact. Basic Research Findings Publications Citations Conference Proceedings Biomarkers Technology Development Inventions Patents Biomarkers Clinical Development Clinical trials Biomarkers Approved Products FDA Approvals of Drugs, Devices and Biologics Public Health Impact Population Results Patient Outcomes Biomarkers 2

American Evaluation Association Annual Meeting, November 2011 Robust, repeatable data linkages support portfolio analysis focused systems Basic Research Findings Publications Citations Conference Proceedings Biomarkers Technology Development Inventions Patents Biomarkers Clinical Development Clinical trials Biomarkers Approved Products FDA Approvals of Drugs, Devices and Biologics Public Health Impact Population Results Patient Outcomes Biomarkers NIH’s electronic Scientific Portfolio Assistant (e-SPA) currently provides linkages to early stage portfolio outputs and impact. 3

American Evaluation Association Annual Meeting, November 2011 Expanding the range of data-driven portfolio analysis NCI’s Office of Science Planning and Assessment is conducting analyses of additional outputs such as cancer Biomarkers and FDA approved drugs. Basic Research Findings Publications Citations Conference Proceedings Biomarkers Technology Development Inventions Patents Biomarkers Clinical Development Clinical trials Biomarkers Approved Products FDA Approvals of Drugs, Devices and Biologics Public Health Impact Population Results Patient Outcomes Biomarkers 4

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of cancer biomarkers? Background Feasibility study to link funding data to breast cancer biomarkers Thomson Reuters’ Integrity platform includes Biomarkers - High quality, manually curated data - Coverage of various cancer disease areas Linkage to NCI funded projects through supporting reference material - Research publications considered by the Integrity editorial team as important for development of breast cancer biomarkers. - Research publications acknowledging NCI project support 5

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of cancer biomarkers? Methodology Patent Indication Biomarker Acknowledgement of NCI Support Publication Validity Status Integrity Biomarkers Data NIH Projects (IMPAC II) 924 Biomarkers 2,040 Biomarkers with different validity statuses for different breast cancer indications (e.g. in situ ductal carcinoma) 6

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of cancer biomarkers? Findings SOURCES: Thomson Reuters Integrity, ScienceWire, NIH IMPAC II (May 2011) 32.4% 54.4% 32.0% 59.5% 66.7% 7

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of cancer biomarkers? Findings, continued Breast Cancer Biomarker Validity Status Supporting NCI Projects Identified Recommended / Approved92 Late Studies in Humans208 Early Studies in Humans238 Experimental239 Emerging198 TOTAL UNIQUE PROJECTS456 Grant Mechanisms Included R grants = 241 (181 are R01s) U and N grants= 100 P grants = 68 Z grants = 10 Training (F,K,L,T) = 35 8

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of cancer biomarkers? Portfolio Implications Feasibility study has demonstrated: The utility of linking NCI projects to biomarker outputs The role of NCI support in breast cancer biomarker development New portfolio management questions can now be asked, such as: What mechanisms are most effective at supporting the various biomarker stage development? What is the total funding investment made for biomarker development? What diseases lack biomarkers and should be considered for targeted support? 9

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of FDA approved drugs? Background NCI Project [IMPACII] NCI Project [IMPACII] Patent Acknowledgement of NCI Support NDA-Listed Patent [USPTO Patent] NDA-Listed Patent [USPTO Patent] Approved Drugs [FDA Orange Book] Approved Drugs [FDA Orange Book] FDA products linked to NCI funding through acknowledgment in patents listed in the New Drug Application (NDA). Direct patent acknowledgements represent direct links to the NCI projects. Direct acknowledgements by NDA listed patents identified 6 FDA approved drugs linked to 6 NCI funded projects. SOURCES: Thomson Reuters Web of Science, MEDLINE, ScienceWire, The Patent Board, NIH IMPACII, FDA Orange Book, USPTO Issued Patents 10

American Evaluation Association Annual Meeting, November 2011 Publication [MEDLINE] Publication [MEDLINE] Acknowledgement of NCI Support Non-Patent Reference List How has the NCI supported the development of FDA approved drugs? Methodology Links established through non-patent references found in the patents listed in the NDA. High confidence, direct links include those with the same personnel and organization in: NDA listed patent Publication it cites and NCI Project that funded the publication. Automated approach identified high confidence, direct links: 9 drugs linked to 12 NCI funded projects. NCI Project [IMPACII] NCI Project [IMPACII] Patent Acknowledgement of NCI Support NDA-Listed Patent [USPTO Patent] NDA-Listed Patent [USPTO Patent] Approved Drugs [FDA Orange Book] Approved Drugs [FDA Orange Book] SOURCES: Thomson Reuters Web of Science, MEDLINE, ScienceWire, The Patent Board, NIH IMPACII, FDA Orange Book, USPTO Issued Patents 11

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of FDA approved drugs? Findings Drug InformationDrug-Publication-Project RelationshipGrant and Publication Data Trade Name Therapeutic Class Patent Inventor, Publication Author & Project PI Patent Assignee, Publication Affiliation & Project Organization Project Start- Publication - Patent Filing Grant Type JournalCitations Cetrotide Gonadotropins, NEC Schally, AndrewTulane University R01PNAS86 Clolar Montgomery, JohnSouthern Research Institute P01J. Med. Chem.28 Folotyn Antineoplastic Agents, NEC Sirotnak, Francis Sloan-Kettering Institute for Cancer Research R35 Cancer Chemoth. Pharm. 37 Thyrogen Kourides, Ione Sloan-Kettering Institute for Cancer Research R01 Recent Prog. Horm. Res. 61 Trisenox Antineoplastic Agents, NEC Warrell, Raymond Sloan-Kettering Institute for Cancer Research R03Blood187 Zinecard Speyer, JamesNew York University P30‡ N. Eng. J. Med.292 Zolinza Antineoplastic Agents, NEC Marks, Paul Sloan-Kettering Institute for Cancer Research 1966 – P30PNAS34 This table lists 7 of the 12 projects identified as having shared both person and organization information. Orange highlights indicate drug-to-project links established only through non-patent references (no direct patent reference to these projects) ‡ P30CA16087 project was also identified as contributing to one of the breast cancer biomarkers SOURCES: Thomson Reuters Web of Science, MEDLINE, ScienceWire, The Patent Board, NIH IMPACII, FDA Orange Book, USPTO Issued Patents 12

American Evaluation Association Annual Meeting, November 2011 How has the NCI supported the development of FDA approved drugs? Portfolio Implications This analysis has shown that: Non-Patent Reference data can serve as a source for additional linkages to outputs like FDA approved drugs Metadata about the outputs can be effectively used to identify high- confidence, direct links Understanding NCI funding directly related to drug development helps to: Measure late-stage impact of research funding on public health by enabling analysis of drug-specific impacts Potentially identify areas where support is needed for particular indications or therapy targets 13

American Evaluation Association Annual Meeting, November 2011 What can be done when the portfolio to be analyzed is not associated with output data? The International Cancer Research Partners includes 50 international funding agencies focused on cancer research who awarded over 36,000 research projects from Although some members have developed systems for tracking outputs arising from ICRP research awards, the partnership does not yet have a comprehensive system. OSPA carried out a feasibility study of linking research publications in Thomson Reuters Web of Science to research projects in the ICRP portfolio. 14

American Evaluation Association Annual Meeting, November 2011 Publication Outputs Missing Because Award Codes Were Insufficient to Match The feasibility study found that researchers were not acknowledging support by grant number (award code). 5%2%10%9%1% Total ICRP Awards Awards Matched to WoS Publications Percentage of Total ICRP Awards Matched to WoS Publications 15

American Evaluation Association Annual Meeting, November 2011 Additional Matching Methods for Increasing Publication Output Identification Using Funding Agency Names  Successful for funding agencies whose portfolio is exclusively focused on cancer research (e.g. American Cancer Society), but not for broadly-focused portfolios (e.g. Wellcome Trust) Using Principal Investigator Name SOURCES: Thomson Reuters Web of Science and ICRP 16

American Evaluation Association Annual Meeting, November 2011 Identifying Candidate ICRP members using ICRP-funded Publications RankFunding OrganizationPubs 1 National Natural Science Foundation of China Ministry of Education, Culture, Sports, Science and Technology of Japan German Research Foundation506 4 European Union494 5 Italian Association for Cancer Research489 RankFunding OrganizationPubs 1 Italian Association for Cancer Research489 2 Swedish Cancer Society429 3 German Cancer Aid361 4 Danish Cancer Society189 5 French Cancer Research Association178 RankFunding OrganizationPubs 1 Japan Society for the Promotion of Science354 2 Pfizer239 3 Novartis216 4 French Cancer Research Association178 5 Leukemia & Lymphoma Society177 Most Common Cancer-specific Funders Most Common Funders Most Common Non-government Funders 17

American Evaluation Association Annual Meeting, November 2011 Future development of outputs data for portfolio analysis Improve data collection and automation of weighted linkages. Establish data-driven linkages between funding and later stage research outputs and outcomes. Enable portfolio analysis across a broad range of data sources and methodologies to reflect the broad range of research outputs. 18

American Evaluation Association Annual Meeting, November 2011 Acknowledgements Biomarkers - NCI OSPA: James Corrigan, Elizabeth Hsu, Lawrence Solomon - Discovery Logic/Thomson Reuters: Sandeep Patel, Tina Brust FDA Patent Analysis - NCI OSPA: James Corrigan, Elizabeth Hsu, Lawrence Solomon - Discovery Logic/Thomson Reuters: Duane Williams - The Patent Board : Kim Hamilton ICRP - NCI OSPA: James Corrigan, Elizabeth Hsu, Lawrence Solomon, Samantha Finstad (AAAS Fellow) - Discovery Logic/Thomson Reuters: Sandeep Patel, Lauren Taffe 19