Staying Ahead of The Curve: Rethinking Research Performance Yukun Harsono Vice President Product Marketing IREG-4 Astana, Republic of Kazakhstan

2 LEAN TIMES ARE HERE

3 THE NEW REALITY…

4 OTHER FORCES THAT SHAPE LEAN RESEARCH Trend Exacerbated by Economic Downturn

GOVERNMENT POLICIES ON RESEARCH Research ProgramSponsor Description Governments are actively guiding their national research agenda  Assess the quality of research in universities and colleges in the UK  Enable funding bodies to determine how to allocate grants across research projects  Detail by institution and by discipline those areas that are internationally competitive, together with emerging areas  Identify thematic domains for future European support  Part of EU’s strategy to become “the most dynamic competitive knowledge-based economy in the world”

6

SCOPUS POWERING PERFORMANCE ANALYSIS 7 CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF. “The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.” CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF. “The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.” "Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries." Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non- English language journals has led to better results from many countries where English is not the first language RESEARCH ASSESSMENT EXERCISE HIGHER EDUCATION FUNDING COUNCIL FOR ENGLAND (HEFCE) Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language "Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries." CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF. “The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.” CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF. “The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”

8 AUSTRALIAN RESEARCH FRAMEWORK Publication Data for National Assessment  Elsevier provides citation information for the Excellence in Research for Australia (ERA) initiative Cluster One evaluation  Refine affiliation data: “In order for the ARC to process publication records for citation analysis, all publication outputs must be submitted with a Scopus unique article identifier. This allows the ARC to match the citation numbers for each publication outputs quickly and accurately post-submission.”

9 GLOBAL RESEARCH LANDSCAPE CHANGING Countries ranked by published output in 2007

10 GLOBAL RESEARCH LANDSCAPE CHANGING Countries ranked by output growth Percent

11 RESEARCH RESPONDS TO GLOBALIZATION Research now...  Crosses national boundaries  Exhibits high mobility of resources, people, ideas, technologies and infrastructure  Overarches multi-jurisdictional regulation  Responds and reacts to public and private pressures  Networks energetically in real and virtual space/time  Balances on the knife edge of Competition and Collaboration Source: Prof. Daryl Le Grew, University of Tasmania

12

RESEARCH MORE EXCITING BUT ALSO MORE CHALLENGING Q: How many hours per week searching and gathering information? 13 Researchers spend more time looking for information than analyzing and applying it Q: How many hours per week organizing, analyzing and applying information Source: 2007 Survey by Outsell Inc. on 6,300 knowledge workers A:

FUNDING PRESSURES Q: What do you estimate is the approval rate for National Science Foundation grant applications by new researchers? 14 Source: American Academy of Arts and Sciences ( Competition for funding is intense and will continue to intensify Q: What do you estimate to be the average age when biomedical researchers receive their first grant from the National Institute of Health? A:

15 Increased focus on EFFECTIVENESS & EFFICIENCY IMPROVE PERFORMANCE across all workflows that make up the research enterprise

16 RESEARCH EXECUTIVES: KEY CHALLENGES  Need constant flow of information and intelligence about internal performance, benchmarks and comparative performance  Require matrix of tools linking: - Reputation and Ranking - Growth and Development with - Research Performance - Research Strategy Implications  “Have my strategic decisions been effective?”  “Are we capitalizing on new hot areas e.g. stem-cells?”  “Where should my strategic focus be?”  “Who are my true competitors per competency?”  “How is my competency portfolio performing?” Questions Source: Prof. Daryl Le Grew, University of Tasmania

HOW WE MEASURE TODAY  Total number of papers  Total number of citations  Average number of citations per author  Average number of papers per author  Thomson Reuters’ Impact Factor  Bergstrom’s Eigenfactor  Egghe's g-index  SCImago’s Journal Rank  Hirsch's h-index 17

18 SAMPLE MEASURE: PUBLICATION OUTPUT Drawbacks  Bias against smaller institutions  Quantity ≠ Quality

19 SAMPLE MEASURE: REFERENCE VS OUTPUT Drawbacks  Inter-disciplinary differences  Not granular enough

20 NEW MEASURE FOR A NEW REALITY New bottom-up approach needed for identifying scientific leadership EXISTING APPROACH TO PERFORMANCE MEASUREMENT ISSUES Journal-based classification  Assign journals to categories Science categorized into small number of fields and subfields (i.e. highly aggregated classification system)  Example: National Science Foundation groups science into 13 fields and 127 subfields Number and scope of journals too limiting Risk: Too few journals included in analysis & potential for English-speaking bias in journal selection Level of aggregation too high-level Risk: Limited view of performance data on the level where it counts – that of school or large focused laboratory Inter-disciplinary boundaries changing Risk: Not understanding how different disciplines interact with one another on the institutional level Risk: Not seeing new emerging research areas

21

A NEW APPROACH 22 Partitioning the literature This literature is partitioned into ~40,000 highly detailed subject areas. Data We start with the Scopus literature database, which indexes over 1.2 million articles per year, and covers a broad set of scientific areas. Brain Research Biotechnology Earth Sciences Math & Physics Chemistry Biology Infectious Disease Medical Specialties Health Services Social Sciences Engineering Computer Science & EE Humanities # of papers

23 Additional information Strengths can be identified for:  Universities  Regions  Nations Details about strengths:  Market shares  Growth rates  Top competitors  Top researchers Identifying areas of strength

24

DISTINCTIVE COMPETENCY #26 25 Math & Physics Chemistry (1) Earth Sciences Biology Biotechnology Infectious Disease (1) Medical Specialties (27) Health Services (1) Brain Research (2) Social Sciences (1) Engineering Computer Science Humanities Description: Clinical cancer research Molecular endocrinology Molecular neuroscience

DISTINCTIVE COMPETENCY #26 26

THE ADVANTAGES? Bottom-UpCross-SilosGranular Identify “Emerging Areas” Identify “Super Connectors” Identify “Rising Stars” 123

28 IDENTIFYING NATIONAL STRENGTHS Source: Klavans & Boyack, “US vulnerabilities in science and engineering,” 10th International Conference on Science and Technology Indicators, Vienna, Austria, Sept , 2008 Strengths – Top 40 Nations Strengths – USA US strengths (transparent) are overlaid here on the strengths (dark) from another 40 nations Strengths – Overlaid

29

ELSEVIER? We own 25% of the world’s scientific literature and we are usually perceived as simply a STM publisher… But the reality is that We have been indexing and analyzing the world’s research output over the last few years, not just Elsevier-owned content We have been spotting and building new advanced tools on top of our multi-publisher citation dataset to help universities take the guesswork out of research performance assessment We solicited help from key partner institutions to co-define what performance intelligence needs to bring to light in order to help drive success for universities

SUMMARY  Efficiency and effectiveness are realized by taking a deeper look at the research activity workflows and identifying improvement opportunities  Progressive institutions will demonstrate lean research by adopting new performance intelligence tools to identify research strengths to focus on  With more rigorous analytics and better insights into research performance these institutions will be better positioned to gain standing among their peers 31  Economic downturn has intensified focus on research effectiveness and efficiency, bringing a new reality for us all…

Thank You For Listening