Grants Factory How the peer review panel works Mick Tuite (Biosciences) BBSRC/Leverhulme Trust Simon Kirchin (SECL) AHRC Jan 2013

How the Peer Review Panel Works.... Q1:Who has submitted a grant?  BBSRC  Leverhulme Trust  AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

How the Peer Review Panel Works.... Q1:Who has submitted a grant?  BBSRC  Leverhulme Trust  AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

How the Peer Review Panel Works.... Q1:Who has submitted a grant?  BBSRC  Leverhulme Trust  AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

How the Peer Review Panel Works.... Q1:Who has submitted a grant?  BBSRC  Leverhulme Trust  AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

How the Peer Review Panel Works....  Submission to decision  How are fund/not fund decisions reached?  Scoring and ranking  How to influence the panel....legally  Getting on a panel  Panel members – what to expect....!

Time is at a premium for referees and panel members “How much time do I have?” e.g. AHRC: One day meetings – 09:30 to 16:30 25 large project grants 300 minutes 12 minutes per grant! You have limited time ‘in the frame’.... BBSRC e.g. BBSRC Two day meetings 100 large project grants 600 minutes 6 minutes per grant

Process: from submission.....to decision Fundingagency Referees DiscussRank Internal Peer Review Panel ?

Composition of grant-awarding panel Academics Non-academic ‘specialists’ ‘end users’ Admin staff ‘Observers’ Chair Panel secretary

Run up to panel meeting: timelines ~ 4 months: membership of panel confirmed lay summaries sent to panel members grants assigned to panel members referees identified (applicant/member/funder) potential ‘conflicts of interest identified ~3 monthsreferees contacted ~ 3 weeks panel members receive the following: BEFORE - all grants meeting - all referees reports - all responses to referees comments - previous grant record

~ 4 months: membership of panel confirmed lay summaries sent to panel members grants assigned to panel members referees identified (applicant/member/funder) potential ‘conflicts of interest identified ~ 3 monthsreferees contacted ~ 3 weeks panel members receive the following: BEFORE - all grants meeting - all referees reports - all responses to referees comments - previous grant record Run up to panel meeting: timelines

~ 4 months: membership of panel confirmed lay summaries sent to panel members grants assigned to panel members referees identified (applicant/member/funder) potential ‘conflicts of interest identified ~ 3 monthsreferees contacted ~ 3 weeks panel members receive the following: BEFORE - all grants meeting - all referees reports - all responses to referees comments - previous grant record Run up to panel meeting: timelines

Role of the ‘IM’ (Introducing Members)  Two IMs per grant (one leads)  One nominated to lead  Almost certainly not be experts in your ‘field’  Make individual recommendations (numerical score or yes/no)  Open to discussion (....not always)  Chair coerces consensus decision (rank/score)Introducing Member (IM)

Less strategic importance compared to other proposals Proposal poorly written Insufficient preliminary dataProposal lacked focus Proposal overambitious and unlikely to achieve all of its objectives Proposal lacked detail Work programmes poorly integrated Lack of relevant expertise Under resourcedPoor track record Will not significantly advance the fieldPoor value for money Will not significantly increase knowledge in the field Costs not adequately justified Is this project of international quality and therefore worthy of funding? Yes/No What are the IMs looking for? e.g. BBSRC “feedback form” In order of priority, please specify the STRENGTHS of the proposal: In order of priority, please specify the WEAKNESSES of the proposal:

Ranking grants: where is the cutoff ? Smith Jones West North South East Brown Green Black Butcher Baker.. Fund Not fund Modulators: New investigator Industrial contribution Strategic relevance Impact plans Good final report scores Strong publication record Rank order Drawing the line: Budget rules! The ‘grey zone’

The scoring system: panel e.g. BBSRC, Wellcome Trust Score definitions 0 = Invited resubmission 1 = Flawed 2 = Not competitive 3 = Fundable but... 4 = Good 5 = Very good 6 = Excellent 7 = Outstanding/world class 20% “Discuss” “Not fund Fund

The scoring system: referees  Exceptional (fundable)  Excellent (fundable)  Very Good (fundable)  Good (fundable)  Not Competitive (not fundable)  Unfundable (not fundable) = in with a chance... = little chance... = not a chance... (20%) (60%) (20%) INTERNATIONAL QUALITY SCIENCE

Scoring ‘impact’ of research Pathways to impact (PtI) document: “What will be done to ensure that potential beneficiaries have the opportunity to engage with this research?” SCORING SYSTEM  Excellent  Satisfactory  Unsatisfactory How will the impact scores be used? -Taken into account when rank ordering - Feedback to applicants - Not used to reject outstanding applications

Influencing the panel....legally.... Profile – they know who you are  Research/conference papers  Seminars/conferences Presentation of grant  12pt Arial (11pt allowed), layout  Use of images (unpublished data)  Proofread!  Avoid extensive self-citation Lay/technical summary All elements of the form are important Response to referees  Succinct, courteous  Include new information

Influencing the panel....legally.... Profile – they know who you are  Research/conference papers  Seminars/conferences Presentation of grant  12pt Arial (11pt allowed), layout  Use of images (unpublished data)  Proofread!  Avoid extensive self-citation Lay/technical summary All elements of the form are important Response to referees  Succinct, courteous  Include new information

Lay/technical summary What is it? Lay: Describe the proposed research in simple terms in a way that could be publicised to a general audience [up to 4000 chars] Technical: Describe the proposed research in a manner suitable for a specialist reader. [up to 2000 characters] Why is it important? Read by all members of the panel Can attract ‘non-IMs’ to take an interest in your grant What it should do? Make it clear - why this project is interesting - why the project must be funded - what the wider ‘impact’ of the project will be

How do you get on a panel ? Good funding record Good refereeing record  on time  sensible comments Personal contacts Personal contacts  panel members  staff of funding agency A pply!

Outline Stage Reviewed by 1 member of the Advisory Panel (35 academics) Decision either full application or reject Member suggests four referees if recommend full application Decision sent to the Director who approves or rejects them Full application stage Sent to 4 referees, 2 suggested by you and 2 selected by Trust member (final choice Not made by Advisory Panel member) Based on the reviews the Trustees * make the final decision. Leverhulme Trust: How grants are assessed * Footnote: Trustees are mainly current/retired Directors of Unilever

The panel members…who are they? What you hope for..... Informed  Informed – knows your research area No bias  No bias – institution or research area Can identify strengths  Can identify strengths (but not so good on weaknesses...?) Listens  Listens to others – and takes on board their comments Makes clear decisions/recommendations  Makes clear decisions/recommendations Dr “Perfect”

Dr “Know it all” Has an opinion on all grants Seems to know a lot....but actually knows very little Dr “On the ball” Everyone listens to what she/he says Actually does know a lot about a lot...and everyone agrees with his/her decision Dr “Methodical” Has read all the grants (not just ‘case for support’) Can recall fine detail e.g. year applicant got PhD....but unable to make an informed decision.... The panel members…what you get!!

Dr “Elitist” Thinks all projects fundable....provided the PI is from.....Oxford.....Cambridge, or.....Imperial Dr “Its all crap” Finds nothing of value in any grant Makes rude comments about PI and no faith in referees’ integrity Dr “Confused by it all” Struggles to get to grips with any of the grants Has more questions than opinions....easily persuaded by others ?

Dr “Disorganised” Still reading his/her grants during the meeting Left key (slanderous) referees’ comments on train Chaotic presentation as IM Dr “Gullable” Believes that the applicant knows what he/she is talking about....the referees are always right....the chair (or anyone else) can override his/her decision The Clown Little input to discussion – other than jokes Can see the funny side of referees’ comments Liked by everyone...comments ignored...

Being on a grants panel……what to expect 1. Hard work: 4-20 grants per panel meeting 2. You get paid! 3. What makes a grant fundable 4. Disillusioned: many good grants are not funded 5.Exposure to a wide range of topics/fields confidence in the peer review system

The key take home points……. 1. You have little time to make an impact! 2. Get your work known by panel members 3. Pay particular attention to the lay and technical summaries 4. Do not underestimate the role/importance of the panel members 5. Not all grants in the ‘funding zone’ on the day... are actually funded 6. If you really want to know what goes on – get yourself onto a panel!

Some points for discussion……. Q: Is this a fair and transparent process? Q:What are the alternatives? - hypothetical real... Q: Do we need referees and panels? Q:What value the internal peer review system? Q. Any funder-specific questions?