1. Ilya Ponomarev 1, Pawel Sulima 1, Jodi Basner 1, Unni Jensen 1, Joshua Schnell 1, Karen Jo 2, and Nicole Moore 2 A New Approach for Automated Author Discipline Categorization and Evaluation of Cross-Disciplinary Collaborations for Grant programs ilya.ponomarev@thomsonreuters.com 1 Custom Analytics, Rockville, MD 2 National Cancer Institute, Bethesda, MD 10/16/2013 5:30 PM

2. Why Cross-disciplinary Research? 2 “ Interdisciplinary research can be one of the most productive and inspiring of human pursuits” Facilitating Interdisciplinary Research National Academy of Sciences, 2005 Innovation increasingly occurs at the boundaries of disciplines Complex “Puzzles” require diverse background Data avalanche from multiple sources requires fusion of information Convergent technologies require integration across disciplines

3. US Government Funding of Cross-disciplinary R&D 3 DOD DOE NSF NIH NASA

4. How to Measure Success of Cross-disciplinary Program? THIS TALK: 1.In order to measure cross-disciplinarity define disciplines as accurate as possible 2.General approach of automatic assigning grant specific categories to papers and people 3.Application to NCI PS-OC grant program classification? 4 See also J. Basner, “Evaluating Collaboration and Outcomes of Health Research” Friday, 10/18/2013, 11:00am at Gunston East Rm

5. NCI Physical Sciences-Oncology Centers 5 12 centers, 250 Researchers 09/2009-Current InstituteFacilitateGenerate

6. Evaluation: Birds View 1.Use publications as a proxy of outcome 6 2006-2008: 3,367 pubs 2009-2012: 601 reported pubs 2.Compare baseline data set (2006-2008) with ongoing research data set (2009-2012) Web of Science+ Medline 166 active PS-OC investigators 202,000 references 4,199 journal titles productivity impact collaboration Fields convergence J. Basner, Friday, 10/18/2013

7. Evaluation: Birds View Approach: 7 Perform mapping of WoS subject categories to PS-OC categories Calculate PS-OC categories to each paper Calculate (weights of) research interests for each investigator Validate PS-OC 2/3 broad categories Oncology Physical Sciences Life Sciences

8. PS-OC 3 broad categories Oncology Physical Sciences Life Sciences 266 Web of Science Journal Subject Categories 8 Has Oncology SC Multiple SCs per journals (up to 7) Multidisciplinary (meaningless, but “Science”, “Nature”) Some SCs are already inter-disciplinary LSs dominates after aggregation

9. 22 ESI Subject Categories 9 One SC per journal Does not have Oncology Multidisciplinary SC exists also Clinical medicine? LSs dominates after aggregation

10. Mapping. Challenges Approach: 1.Intermediate map on extended 6 Broad Categories 2.Paper level SC assignment based on references 10 PS-OC 3 broad categories Oncology Physical Sciences Life Sciences Web of Science 266 Journal SCs Web of Science 22 Broad ESI categories One SC per journal Does not have Oncology Multidisciplinary SC exists also Clinical medicine? LSs dominates after aggregation Has Oncology SC Multiple SCs per journals Multidisciplinary Some SCs are inter-disciplinary LSs dominates after aggregation

11. Step 1. Introduce 6 Intermediate PS-OC Categories for Better Selection: 11 PS – Physical Sciences LS – Life Sciences OC – Oncology MED – Medicine OTH – Others MULT – Multidisciplinary 11 (very often MED journals are closer to ON than LS) Will be dropped on final stage

12. Step 2. Map 265 WoS JSC to 6 PS-OC Categories: 12 Examples: a) Obvious: Acoustics  PS, Chemistry, Analytical  PS Oncology  OC, Management  OTH b) Dominant: Biophysics  PS c) Dominant: Physics, Multidisciplinary  PS d) Meaningless: Multidisciplinary  MULT (usually published in “Nature”, “Science” or “PNAS”) Meaningless in terms of assignment PS-OC category: article published in MULT journal can be about PS, or about LS, or OC. Usually, it is not interdisciplinary article. Additional re-classification of article’s research field is needed based on references.

13. Step 3. Assign PS-OC Categories Weights to Each Journal 13 (Journals in WoS can have 1 or 2, or 3, … even 7 SCs)  Examples: Journal “Radiation Research” – 3 SCs: Biology  LS Biophysics  PS Radiology, NM  PS LS PS Map Select distinct PS-OC categories 2 Count total (denominator ) Weights) LS=1/2 PS=1/2 OC=0 MED =0 MUL=0 OTH=0 Each journal should be counted equally

14. Step 4. Calculate combined J-R weights for publications: 14 Example: Coffey D., Getzenberg R. JAMA, 2006  1 journal cat (MED=1)  26 Refs: 14 Journal weightsAver. Refs Weights LS=0 PS=0 MED=1 OC=0 MUL=0 OTH=0 LS=0.23 PS=0.04 MED=0.17 OC=0.36 MUL=0.19 OTH=0 ½ (Journal + Refs) LS=0.12 PS=0.019 MED=0.58 OC=0.18 MUL=0.1 OTH=0 Better assignment of paper’s field based on information what paper cites

15. Step 5. Collect all publications for each investigator, calculate average weights, and rank PS-OC categories: 15 Example. David A  8 pubs:  Average JR weights Averaged J-R Weights LS =0.32 PS =0.04 MED=0.23 OC =0.41 OTH =0.01 Person Inter- disciplinarity LS =2 PS =4 MED=3 OC =1 OTH =5 Ranks 3

16. Step 6. Redistribute MED and OTH weights between OC,LS, and PS 16 LS =0.32 PS =0.04 MED=0.23 OC =0.41 OTH =0.01 LS =0.4 PS =0.05 OC =0.55

17. Validation 17 At the beginning of the program: Investigators self-nominated themselves as oncologists or physicists

18. Applications: how publication patterns change 18

19. Future Development 19 Physical Scientist Oncologist Life Scientist PS-OC Network InvestigatorsOutside Network Co-authors

20. Conclusions 20 Automated approach for decomposition of scientific publications into grant specific discipline categories Multi-step method with intermediate mapping Weighted SC assignment based on article’s and its references’ SCs Precision-recall validation based on investigators’ self- categorizations Oncologists within the NCI’s PS-OC program are publishing more physical sciences research and physical scientists are publishing more oncology or life sciences research during years of program participation.

21. ilya.ponomarev@thomsonreuters.com Thomson Reuters Custom Analytics Rockville, MD

22. SUPPORTING SLIDES