1. Towards indicators for ‘opening up’ science and technology policy Ismael Rafols, Tommaso Ciarli, Patrick van Zwanenberg and Andy Stirling Ingenio (CSIC-UPV), Universitat Politècnica de València & SPRU —Science and Technology Policy Research, University of Sussex Stockholm, October 2013 Building on work with Loet Leydesdorff and Alan Porter

2. Paper born out of the reflection on the contrast between interdisciplinary research and journal rankings Interdisciplinary maps versus rankings

3. On the role of scientific advice in policy (scientometric is the science of science –hence scientific advice) The linearity-autonomy model of scientific advice (Jasanoff, 2011)  Scientific knowledge is the best possible foundation for public decisions  Scientists should establish the facts that matter independently. –S&T indicators produce evidence of these facts. However, this (enlightenment) model has been challenged  The mechanisms to establish facts and make decisions is a social process –“knowledge enables power, but power structures knowledge” (Stirling, 2012)  Modes of advice: The pure scientist vs. honest broker (Pielke, 2007) What is (should be) the role of STI indicators in policy advice? Closing down vs. Opening up

4. The challenge Problems with current use of S&T indicators Use of conventional S&T indicators is en *problematic* (as many technologies, in particular those closely associated with power, e.g. nuclear)  Narrow inputs (only pubs!)  Scalar outputs (rankings!)  Aggregated solutions --missing variation  Opaque selections and classifications (privately owned databases)  Large, leading scientometric groups embedded in government / consultancy, with limited possibility of public scrutiny  Sometimes even mathematically debatable  Impact Factor of journals (only 2 years, ambiguity in document types)  Average number of citations (pubs) in skewed distributions

5. From S&T indicators for justification and disciplining… Justification in decision-making Weak justification, “Give me a number, any number!” Strong justification, “Show in numberrs that X is the best choice!” S&T Indicators have a performative role:  They don’t just measure. Not ‘just happen to be used’ in science policy (neutral)  Constitutive part incentive structure for “disciplining” (loaded)  They signal to stakeholders what is important. Institutions use these techniques to discipline subjects  Articulate framings, goals and narratives on performance, collaboration, interdisciplinarity…

6. … towards S&T indicators as tools for deliberation Yet is possible to design indicators that foster plural reflection rather than justifying or reinforcing dominant perspectives This shift is facilitated by trends pushed by ICT and visualisation tools  More inputs (pubs, pats, but also news, webs, etc.)  Multidimensional outputs (interactive maps)  Multiple solutions -- highlighting variation, confidence intervals  More inclusive and contrasting classifications (by-passing private data ownership? Pubmed, Arxiv)  More possibilities for open scrutiny (new research groups)

7. 1. Conceptual framework: “broadening out” vs. “opening up” policy appraisal

8. Policy use of S&T indicators: Appraisal Appraisal: ‘the ensemble of processes through which knowledges are gathered and produced in order to inform decision-making and wider institutional commitments’ Leach et al. (2008) Breadth: extent to which appraisal covers diverse dimensions of knowledge Openness: degree to which outputs provide an array of options for policies.

9. Policy use of S&T indicators: Appraisal Appraisal: ‘the ensemble of processes through which knowledges are gathered and produced in order to inform decision-making and wider institutional commitments’ Leach et al. (2010) Example: Allocation of resources based on research “excellence” Breadth: extent to which appraisal covers diverse dimensions of knowledge Narrow: citations/paper Broad: citations, peer interview, stakeholder view, media coverage, altmetrics Openness: degree to which outputs provide an array of options for policies. Closed: fixed composite measure of variables  unitary and prescriptive Open: consideration of various dimensions  plural and conditional

10. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Leach et al. 2010 Appraisal methods: broad vs. narrow & closing vs. opening

11. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Appraisal methods: broad vs. narrow & close vs. open cost-benefit analysis open hearings consensus conference scenario workshops citizens’ juries multi-criteria mapping q-method sensitivity analysis narrative-based participant observation decision analysis risk assessmentstructured interviews Stirling et al. (2007)

12. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Appraisal methods: broad vs. narrow & closing vs. opening Most conventional S&T indicators??

13. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Broadening out S&T Indicators Conventional S&T indicators?? Broadening out Incorporation plural analytical dimensions: global & local networks hybrid lexical-actor nets etc. New analytical inputs: media, blogsphere.

14. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Appraisal methods: broad vs. narrow & closing vs. opening Journal rankings University rankings Unitary measures that are opaque, tendency to favour the established perspectives … and easily translated into prescription European Innovation Scoreboard

15. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Opening up S&T Indicators Conventional S&T Indicators?? opening-up Making explicit underlying conceptualisations and creating heuristic tools to facilitate exploration NOT about the uniquely best method Or about the unitary best explanation Or the single best prediction

16. 2. Examples of Opening Up a.Broadening out AND Opening up b.Opening up WITH NARROW inputs

17. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making 1. Preserving multiple dimensions in broad appraisals Conventional S&T indicators?? Leach et al. 2010 Broadening out opening-up

18. Composite Innovation Indicators (25-30 indicators) European (Union) Innovation Scoreboard Grupp and Schubert (2010) show that order is highly dependent on indicators weightings. Sensitivity analysis

19. Solution: representing multiple dimensions (critique by Grupp and Schubert, 2010) Use of spider diagrams allows comparing like with like U-rank, University performance Comparison tools (Univ. Twente) 5.4 Community trademarks indicator

20. U-Map: Comparison of Universities in Multiple Dimensions

21. 2. Examples of Opening Up b. Opening up WITH NARROW inputs

22. narrow broad closing-downopening-up range of appraisals inputs (issues, perspectives, scenarios, methods) effect of appraisal ‘outputs’ on decision-making Opening up S&T Indicators Conventional S&T Indicators?? Leach et al. 2010 opening-up Making explicit underlying conceptualisations and creating heuristic tools to facilitate exploration NOT about the uniquely best method Or about the unitary best explanation Or the single best prediction

23. Manchester Inn Inst Warwick Business School Disciplinary Diversity of Publications Variety1920 Shannon Entropy 2.9663.078 Disciplinary Diversity of References Variety1720 Shannon Entropy 3.3783.153 Disciplinary diversity of Citations Variety2224 Shannon Entropy 3.4153.503 Interdisciplinarity as diversity

24. Rafols, Porter and Leydesdorff (2010) Cognitive Sci. Agri Sci Biomed Sci Chemistry Physics Engineering Env Sci & Tech Matls Sci Infectious Diseases Psychology Social Studies Clinical Med Computer Sci Business & MGT Geosciences Ecol Sci Econ Polit. & Geography Health & Social Issues A Global Map of Science 222 SCI-SSCI Subject Categories

25. Warwick Business School Subject Categories of publications Nodes labelled if >0.5% publications

26. Manchester MIoIR Subject Categories of publications Nodes labelled if >0.5% publications

27. Heuristics of diversity (Stirling, 1998; 2007) Diversity: ‘attribute of a system whose elements may be apportioned into categories’ Characteristics: Variety: Number of distinctive categories Balance: Evenness of the distribution Disparity: Degree to which the categories are different. Variety BalanceDisparity Herfindahl (concentration):  i p i 2 Shannon (Entropy):  i p i ln p i Dissimilarity:  i d i Generalised Diversity (Stirling)  ij(i  j) (p i p j )  (d ij ) 

28. Manchester Innov Inst Warwick Business S Diversity of Publications Variety 1920 Balance 0.5430.460 Disparity 0.8170.770 Shannon Entropy 2.9663.078 Rao-Stirling Diversity 0.7260.680 Diversity of References Variety 1720 Balance 0.4150.325 Disparity 0.8460.780 Shannon Entropy 3.3783.153 Rao-Stirling Diversity 0.7290.689 Diversity of Citations Variety 2224 Balance 0.5050.454 Disparity 0.8360.801 Shannon Entropy 3.4153.503 Rao-Stirling Diversity 0.7710.736 Comparing degree of interdisciplinarity of two university units: Manchester is more??

29. Multiple concepts of interdisciplinarity: Conspicuous lack of consensus but most indicators aim to capture the following concepts Integration (diversity & coherence) Research that draws on diverse bodies of knowledge Research that links different disciplines Intermediation Research that lies between or outside the dominant disciplines

30. Diversity ISSTI Edinburgh WoS Cats of references Assessing interdisciplinarity

31. ISSTI Edinburgh Observed/Expected Cross-citations Coherence Assessing interdisciplinarity

32. ISSTI Edinburgh References Intermediation Assessing interdisciplinarity

33. Summary: IS (blue) units are more interdisciplinary than BMS (orange) More Diverse Rao-Stirling Diversity More Coherent Observed/Expected Cross-Citation Distance More Interstitial Average Similarity

34. 2. Excellence: Opening Up Perspectives Provide different perspectives of performance (alternative measures of the same type of indicator)

35. Are measures of “excellence” consistent and robust? Good Average Bad Van Eck, Waltman et al. (2013) More basic More applied Clinical neurology Is basic always better than applied ? Citations: not stable to changes in classification and granularity (Zitt et al., 2005; Adams et al., 2008).

36. Measures of “excellence” Which one is more meaningful??

37. The new Leiden ranking (2011-12) Different measures of performance MNC, MNCS, MNCJ, Top 10%, Under different conditions (fractional, language) Include confidence interval (bootstrapping)

38. 3. Summary and conclusions

39. S&T indicator as a tools to open up the debate ‘Conventional’ use of indicators (‘Pure scientist ‘--Pielke)  Purely analytical character (i.e. free of normative assumptions)  Instruments of objectification of dominant perspectives  Aimed at legitimising /justifying decisions (e.g. excellence)  Unitary and prescriptive advice Opening up scientometrics (‘Honest broker’ --Pielke)  Aimed at locating the actors in their context and dynamics  Not predictive, or explanatory, but exploratory  Construction of indicators is based on choice of perspectives  Make explicit the possible choices on what matters  Supporting debate  Making science policy more ‘socially robust’  Plural and conditional advice Barré (2001, 2004, 2010), Stirling (2008)

40. Strategies for opening up or how to “keep it complex” yet “manageable” Presenting contrasting perspectives  At least TWO, in order to give a taste of choice Simultaneous visualisation of multiple properties / dimensions  Allowing the user take its own perspective Interactivity  Allowing the user give its own weigh to criteria / factors  Allowing the user manipulate visuals.

41. Is ‘opening up’ worth the effort? (1) Sustaining diversity in S&T system Decrease in diversity. Potential unintended consequence of the evaluation machine: Why diversity matters Systemic (‘ecological’) understanding of the S&T  S&T outcomes depend on synergistic interactions between disparate elements. Dynamic understanding of excellence and relevance  New social needs, challenges, expectations from S&T Manage diverse portfolios to hedge against uncertainty in research  Office of Portfolio Analysis (National Institutes of Health) http://dpcpsi.nih.gov/opa/ http://dpcpsi.nih.gov/opa/ Open possibility for S&T to work for the disenfranchised  Topics outside dominant science (e.g. neglected diseases)

42. Is ‘opening up’ worth the effort? (2) Building robustness against bias Do conventional indicators tend to favour incumbents? Hypothesis: Elites and incumbents (directly or not) influence choice of indicators, which tend to benefit them… “knowledge enables power, but power structures knowledge” (Stirling, 2012)  Crown indicator –Standard measure of performance (~1990-2010) –‘systematic underrating of low-ranked scientists’ (Opthof and Leydesdorff, 2010) (Not spotted for 15 years!)  Journal rankings in Business and Management. –systematic underrating of interdisciplinary (heterodox) depts. (Rafols et al., 2012).  Others?? H-index?? –favours established academics over younger.

45. ‘lock-in’ to policy favoured by incumbent power structures multiple practices, and processes, for informing social agency (emergent and unstructured as well as deliberately designed ) complex, dynamic, inter- coupled and mutually- reinforcing socio- technical configurations in science narrow scope of attention Conventional Policy Dynamics SOCIAL APPRAISAL GOVERNANCE COMMITMENTS simple ‘unitary’ prescriptions POSSIBLE FUTURES expert judgements / ‘evidence base’ “best / optimal /legitimate” S&T indicators risk assessment cost-benefit analysis also: restricted options, knowledges, uncertainties in participation incomplete knowledges Res. Excellence $ IIIIII GUIDANCE / NARRATIVE Stirling (2010)

46. POSSIBLE PATHWAYS MULTIPLE TRAJECTORIES SOCIAL APPRAISAL GOVERNANCE COMMITMENTS broad-based processes of ‘precautionary appraisal’ ‘opening up’ with ‘plural conditional’ outputs to policymaking dynamic portfolios pursuing diverse trajectories viable options under: conditions, dissonant views, sensitivities, scenarios, maps, equilibria, pathways, discourses multiple: methods, criteria, options, frames, uncertainties, contexts, properties, perspectives Breadth, Plurality and Diversity Sustainability $                 Stirling (2010)

47. S&T indicator as a tools to open up the debate ‘conventional’ use of indicators  Instruments of objectification  Analytical character (i.e. free of normative assumptions)  Aimed at making decisions (e.g. excellence)  Unitary and prescritive advice Opening up scientometrics  Construction of indicators is based on choice of perspectives  implicit normative choice on what matters  Aimed at locating the actors in their context and dynamics  Not predictive, or explanatory, but exploratory  Supporting debate  making science policy more ‘socially robust’  Plural and conditional advice Barré (2001, 2004, 2010), Stirling (2008)

48. Heuristics of diversity (Stirling, 1998; 2007) Diversity: ‘attribute of a system whose elements may be apportioned into categories’ Characteristics: Variety: Number of distinctive categories Balance: Evenness of the distribution Disparity: Degree to which the categories are different. Variety BalanceDisparity Herfindahl (concentration):  i p i 2 Shannon (Entropy):  i p i ln p i Dissimilarity:  i d i Generalised Diversity (Stirling)  ij(i  j) (p i p j )  (d ij ) 

49. Rafols, Porter and Leydesdorff (2010) Cognitive Sci. Agri Sci Biomed Sci Chemistry Physics Engineering Env Sci & Tech Matls Sci Infectious Diseases Psychology Social Studies Clinical Med Computer Sci Business & MGT Geosciences Ecol Sci Econ Polit. & Geography Health & Social Issues A Global Map of Science 222 SCI-SSCI Subject Categories CD-ROM version of the JCR of SCI and SSCI of 2009. Matrix of cross-citations between journals (9,000 x 9,000) Collapse to ISI Subject Category matrix (222 x 222) Create similarity matrix using Salton ’ s cosine

50. Diversity indexes Stirling Generalised Diversity

51. Diversity indexes Stirling Generalised Diversity  =0,  =0 Number of disciplines

52. Diversity indexes : Stirling Generalised Diversity  =0,  =1 Simpson (Herfindahl) Index

53. Diversity indexes Generalised Stirling Diversity  =1,  =1 quadratic entropy

54. Different aspects of diversity are uncorrelated r var,bal = 0.18, p <.001 r var,dis = 0.32, p <.001 r bal,dis = -0.20, p <.001 Yegros et al. (2010) Which diversity measure should we choose?

55. Leiden Ranking: high correlation but important individual differences Spearman‘s rank correlation coefficient matrix. (Thanks to Daniel Sirtes and Ludo Waltman for sharing this data)