Measuring science with meaningful metrics Interdisciplinarity and transdisciplinarity Edwin Horlings 23 januari 2014

2 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Meaningful metrics for evaluation Why do you evaluate? accountability learning What do you evaluate? output impact process What metrics are meaningful?

MONODISCIPLINARY SCIENTOMETRICS

4 | Kenniscoproductie voor de grote maatschappelijke uitdagingen How does a scientist develop his/her portfolio? Empirical evidence for the reward system of science priority seeking: strive to be the first to solve a problem receive recognition from a community of peers Search strategies Search: the process by which an individual scientist identifies, enters, develops, and exits a problem area and its associated community of peers. Strategy: the scientist’s strategic positioning in a competitive environment presumes a degree of planning, coherence and consistency to problem choice over time. Horlings, E., & Gurney, T. (2013). Search strategies along the academic lifecycle. Scientometrics, 94(3),

5 | Kenniscoproductie voor de grote maatschappelijke uitdagingen The portfolio of Ronald Plasterk

6 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Six propositions 1.A scientist’s work consists of multiple finite research trails 2.A scientist will work in several parallel research trails 3.A scientist’s role in research trail selection changes along the lifecycle 4.The start and end of research trails is associated with career changes 5.The start and end of research trails is associated with the potential for reputational gain 6.A scientist’s portfolio will converge before it diverges   Horlings, E., & Gurney, T. (2013). Search strategies along the academic lifecycle. Scientometrics, 94(3),

7 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Proposition 2: Parallel research trails Years since first documented publication Number of parallel research trails PhD Professor Postdoc

8 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Proposition 3: Role in problem selection changes along the lifecycle Percentage of publications written in first, other or last author positions in three phases of the academic lifecycle author positionPhDPostdocProfessor first other last total100 Horlings, E., & Gurney, T. (2013). Search strategies along the academic lifecycle. Scientometrics, 94(3),

9 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Proposition 5: Start and end of trails associated with reputational gain

INTER/TRANSDISCIPLINARY SCIENTOMETRICS

11 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Definitions Interdisciplinary research (IDR) integrates separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem Transdisciplinary research (TDR) integrates knowledge by researchers and stakeholders from different fields, working on a common problem over an extended period of time, developing shared conceptual frameworks, skills, and goals, transcending disciplinary perspectives Wagner, C.S. et al. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), Rosenfield, P. L. (1992) ‘The potential of transdisciplinary research for sustaining and extending linkages between the health and social-sciences’, Social Science & Medicine, 35 (11), Klein, J. T. (2008) ‘Evaluation of interdisciplinary and transdisciplinary research - A literature review’, American Journal of Preventive Medicine, 35 (2), S116-S23.

12 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Integration is the key Involves different knowledge producers disciplines, societal actors Combines heterogeneous knowledge scientific, technical, experiental, social Produces wider variety of outputs peer-reviewed papers, policy reports, tools, etc. Different production processcreating consensus; aligning incentives; managing expectations; etc. New performance criterianew notions of impact and quality

13 | Kenniscoproductie voor de grote maatschappelijke uitdagingen IDR: social network analysis Physics good coverage Sociology poor coverage Computer science conference proceedings

14 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Measuring the characteristics of TDR Funding information: co-financing by stakeholders Collaboration: involvement of stakeholders Output: variety

15 | Kenniscoproductie voor de grote maatschappelijke uitdagingen TDR funding Acknowledgements and funders were correctly retrieved YesNo Authors acknowledge KVK programme funding Yes57 (26%)60 (27%) No66 (30%)20 (9%) N=221; 18 papers (8%) without acknowledgement in the metadata Koier, E., & Horlings, E. (2014). “How accurately does output reflect the nature and design of transdisciplinary research programmes?” Research Evaluation, rvu027 (in press).

16 | Kenniscoproductie voor de grote maatschappelijke uitdagingen TDR collaborators Type of organisation Articles co-authored by at least one author affiliated to a type of organisation Percentage of total articles (n=304) University25183% Public research organisation19363% Government62% Firm279% NGO62% Koier, E., & Horlings, E. (2014). “How accurately does output reflect the nature and design of transdisciplinary research programmes?” Research Evaluation, rvu027 (in press).

17 | Kenniscoproductie voor de grote maatschappelijke uitdagingen TDR outputs Output classified by the project leaders as scientific Scientific papers, peer reviewed 542 Scientific papers, other106 Proceedings115 PhD theses11 Output classified by the project leaders as non-scientific Audio2 Books43 Brochures68 Final project reports171 Media322 Press releases26 Popular articles about science117 Posters159 Presentations1,087 Project factsheets247 Project newsletters81 Reports515 Koier, E., & Horlings, E. (2014). “How accurately does output reflect the nature and design of transdisciplinary research programmes?” Research Evaluation, rvu027 (in press).

18 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Impact Societal impact is the holy grail of TDR Does TDR produce societal impact? Survey among project leaders and participants of 178 projects in Climate changes Spatial Planning ( ) and Knowledge for Climate ( ) Stefan P.L. de Jong et al. (2014). “Exploring the promises of transdisciplinary research: a quantitative study of two climate research programmes” (draft paper)

19 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Different types of impact project results: were used in societal debates contributed to including climate change knowledge in investment decisions created political support for climate adaptation measures helped to postpone or cancel climate adaptation measures produced climate adaptation measures and strategies that were implemented created new contacts improved existing contacts

20 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Each impact requires a custom design Societal debate Investment decisions Political support Implementa -tion Improved contacts New contacts Share of societal actors in the project – Number of informally involved societal actors Prominence of societal actors in the project + + Influence of societal actors on research process + Number of publication types aimed at non- scientific audience

21 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Lessons for the evaluation of TDR Multi-method approach of which scientometrics is just one Calls for new criteria (dealing with variety of knowledge inputs and outputs) Evaluate (project)specific impacts Stronger emphasis on process and design, not just output

Thank you for your attention Edwin Horlings |

23 | Kenniscoproductie voor de grote maatschappelijke uitdagingen Risk versus reward in science reward risk a 1 b 1 a 0 b 0 a 2 b 2 a 0,b 0 Wouter P.C. Boon & Edwin Horlings (2014), Knowledge co- production in protective spaces””, draft paper