20 YEARS OF SCIENTIFIC RESEARCH IN HEALTH/WORK/ENVIRONMENT September 6, 2012 Thoughts of a reviewer Prof Dick Heederik, PhD IRAS, Utrecht University, The Netherlands

Reviewing scientific proposals and programs  Involved in BESLPO project evaluation and the SSD Health program evaluation as panel member

Program evaluation SSD Health  Scientific quality  Networking  Internationalization  Policy relevance  Coverage of the program  Characteristics of the program  Project level (network, budget, duration …)  Program level (calls, budget, …)  Follow-up committee  International projects/EU  Clusters  Dissemination  Relevance of the program  Other considerations

Developments in the scientific community  Scientific production more dominated by teams, even in field traditionally dominated by solists  Teams produce more highly cited papers  Development is seen in all areas, over time, even after removal of self-citations  Networks have become the dominant and most prominent way to go  Research at disciplinary frontiers and in novel areas is often inter-disciplinary  Research management becomes interested in R&D structures

The role of peer review in project and proposal selection?

Criteria voor Quality Assessment in Peer Review (NIH)  Significance impact (does the project address an important problem or critical barrier to progress in the field)?  Investigators (well suited to the project)  Innovation (shift current research or practice paradigms)  Approach appropriate?  Will the scientific environment contribute to succes?

Scientific Quality: publications, citations, publication networks ….. From intuitive interpretation to quantitative analysis …

Quality: characteristics of good research groups  Leaders of high performing research groups survey:  High performance research (publications, citations (normalized for group size)  Stronger research commitment  More effort in group management  Spent more time on network management  All rounders  Verbree et al., Rathenau institute, NL

Quality: different types of excellent groups  Output types correlate poorly: publications, citations, productivity, citations per publication  …, and have different determinants.  So, it also depends to some extent on what is asked  Verbree et al Rathenau Institute

Peer review program evaluation: output evaluation parameters

Program performance ~1.8 Meuro/year internal support, ~ 70% external projects

Impact in different sub-fields

SSD  Too early to make a formal quantitative analysis of impact of the BELSPO Health program  Does this result in unbiased impression given the likely additional funding from other sources?  In essence evaluation of participating groups

Networking and internationalization  Strong interdisciplinary collaboration (PARHEALTH, S2Nano, SHAPES)  Projects did not make use of additional funding possiblities to finance international partners  Some groups had strong international networks but connection with international research community could be strengthened  Collaboration with industry limited (S2Nano)

More formal approaches to analyze networks: 44-cluster co-authorship network of papers at the 10% highly-cited threshold (Rosas et al. PLoSone, 2011 )

Collaborative output

Dissemination  Follow-up committee not for all projects useful, for others effective  Projects which have a stronger basic research focus could benefit from a scientific steering committee  More options for dissemination should be considered (internet databases, software tools, etc.)  To make scientific results available for society may require an additional research cycle

Dissemination  Role in evaluation of future exposure standards (PARHEALTH)  Results can be used by local planners (cost benefits of various modes of transport SHAPES)  Use of developed concepts in testing guidelines (S2Nano)  Breakthrough technology (ANIMO)

Coverage of the field  20% of diseases associated with environmental factors.. (Kirsh-Volders et.al. 2012)  Occupational exposures (chemical, biological, physical)  Environmental exposures (outdoor, indoor)  Do we know the priorities in our field (risk, impact, time, DALY)?

The field environment and health  Small populations at risk, high risks (MICATR)  Large populations at risk, low risks (SHAPES, PARHEALTH)  New emerging risks (S2Nano)  New approaches/technologies (MIC-ATR, ANIMO)

Overall appreciation  Small program  Relevant for capacity building in Belgium  Relevant for public health in relation to the environment in Belgium  The program delivers value for money  Effect of most projects is beyond the project period

Environment and health: funding  Public health  Public funding versus industry funding  Mixed funding (Health Effects Institute)?

Where are we going?