1. Dalian 21-23 May 2012 The role of peer review in the selection of academic talent Pleun van Arensbergen

2. 21-23 May 2012 Pleun van Arensbergen | 2 | Variation in peer review -Peer review is used for different purposes, involving various objects, e.g. scientific papers, research proposals, research programs, grant applications. -Considered to be ‘the’ system of quality control for and within the scientific community, although contested -Also used for selection of academic talents

3. 21-23 May 2012 But does it work? Previous research on applications for personal grants shows that peer review scores do not predict application success (Van den Besselaar & Leydesdorff 2009) What is the role of peer review in talent selection? Quantitative study of the review procedure of 905 grant applications from 2009 Innovational Research Incentives Scheme Pleun van Arensbergen | 3

4. 21-23 May 2012 Dutch Personal Grant Program -Personal grants for excellent researchers belonging to the top 10-20% of their population, in three career phases Early career (max €250.000) Intermediate career (max €800.00) Advanced career (max €1.500.000) -Growing importance of funding program -Increasing number applications, decreasing allocation rate -Training and preselection at universities Pleun van Arensbergen | 4 |

5. 21-23 May 2012 Selection procedure Panel criteria -Quality of applicant -Quality of proposal -Research impact 1540 No data 905 635 (41%) 552 353 (39%) 263 289 (52%) 17 % granted

6. 21-23 May 2012 Talent identified by peers? -No clear identification of talent Pleun van Arensbergen | 6

7. 21-23 May 2012 Role of peer review in selection (1) -Moderately strong correlation between peer review and panel review (=.52) -Strongest correlation with panel review of proposal (=.55 /.64) -Weakest correlation with panel review of research impact (=.22 /.36) -If selection would have been based on peer review: 17% of current interview candidates would not have been selected for interview, 24% not eventually granted Pleun van Arensbergen | 7

8. 21-23 May 2012 Predicting talent selection Model constructed with logistic regression analysis, including Only constant: 54% correctly predicted Peer reviews: 64% correctly predicted All reviews: 83% correctly predicted Peer reviews and research impact not included Only panel review of proposal (B= 1.86) & researcher included (B= 1.40) Minor role of peer review in selection of talent: not the experts on topic decide, but panel Pleun van Arensbergen | 8

9. 21-23 May 2012 Peer versus Panel Peer review as part of selection process First (large) selection made without peer review! Peer review versus panel review: Peer review: experts on the topic individually review one application, send back their review Panel review: good researchers in the discipline review all or a set of applications, discuss their review within the panel and decide on a ranking Level of ‘peerness’ Pleun van Arensbergen | 9

10. 21-23 May 2012 Panel selects (Based on 30 interviews with panel members) Excellent CV (e.g. publications, awards, grants, international experience) Can be reviewed for everyone, also by non-peers Do I understand the proposal? Should be understandable for everyone, also for non-peers Is it innovative and feasible? Using (and reviewing) external peer reviews Is it their own research idea / line? Checked during the interview Motivation, passion and enthusiasm Checked during interview Pleun van Arensbergen | 10

11. 21-23 May 2012 Importance of interview Pleun van Arensbergen | 11 -clarification -own idea -motivation & enthusiasm But also -communication skills -personal attitude -having a ‘click’ Many talented researchers, so selection partly dependent on coincidence, but “real talents are easy to identify”

12. 21-23 May 2012 Talent identification in ‘real life’ Which talents do we want to select for our own department? -Good CV -Motivation & enthusiasm -Fit within the group -Expertise -Social characteristics (e.g. nice and helpful person, a ‘good colleague’) Often talents are not recognized directly, but after working with them for some time Pleun van Arensbergen | 12

13. 21-23 May 2012 Recommendations Provide peers with a format for their review, including a clear motivation of their argument Maybe only for evaluation of innovativeness and feasibility Let the panel only identify the top and bottom, the middle part will be randomly allocated Top applicants are funded directly Applicants in grey zone are randomly selected Do not standardize but use a variety of selection procedures Pleun van Arensbergen | 13

14. 21-23 May 2012 p.vanarensbergen@rathenau.nl

15. 21-23 May 2012 Predicting interview selection Model constructed with logistic regression analysis, including Only constant: 60% correctly predicted Peer reviews: 69% correctly predicted All reviews: 77% correctly predicted Peer reviews and research impact not included Only panel review of proposal (B= 1.36) & researcher included (B= 0.71) Pleun van Arensbergen | 15