ICT Programme Operations Unit Information and Communications Technologies How to fill in the IER form ICT Calls 2013

This tutorial gives guidelines on how to carry out the “remote” stage of an ICT evaluation How to assess the proposal Scoring the proposal Filling in and returning the Individual Evaluation Form (IER) Introduction

You have signed a non-disclosure agreement Before, during and after this evaluation you do not disclose any information about the proposals which you have seen Keep any printed copies secure when not in use; bring them with you to Brussels You are the evaluator Do not pass this responsibility to anybody else Do not discuss the proposals with anybody else But first, remember your responsibilities

Do not contact the proposers Evaluate only the proposal as it was submitted to us at the call deadline, without any later additions or clarifications from the proposer The identities of evaluating experts are never revealed to the proposers concerned Declare any potential conflict of interest If there is a situation which might prevent you evaluating a proposal impartially, click on “I cannot evaluate this proposal” (see Rivet manual page 9) Your responsibilities

For the remote stage of this evaluation we are using a software tool “borrowed” from elsewhere in the Framework programme – Rivet In Rivet the Individual Evaluation Report (IER) is called the Individual Assessment Report (IAR), but the meaning is the same Save your work regularly, unsaved work is lost if you are idle too long and Rivet times you out. If you are reading, thinking or even typing Rivet believes you are idle. Only saving is considered to be an activity. Save your form at least every 20 minutes ! The Rivet tool

Proposals are evaluated on three criteria only Scientific and technical quality Implementation Impact Assess the proposal in terms of all three criteria Each criterion is more fully defined by descriptive “bullet points”, adapted to the project type. These are shown on the evaluation form Provide a comment for each of the “bullet points” (but the bullet points themselves are not scored individually or separately) The Evaluation Criteria

First develop your comments on each criterion ……then select scores accordingly Each criterion is given a score out of five, corresponding to the explanatory comments A threshold of 3/5 is applied on each criterion An overall score is calculated for each proposal by simple addition A threshold of 10/15 is applied on the overall score Out of scope proposals are given low scores on Criterion 1 “Scientific and technical quality” (FET objectives have different thresholds and a weighting scheme; See the FET Proactive evaluation forms in the “Guidance notes for evaluators”) The scoring scale

Use the full scale! Half marks may be given 0 -The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information 1 -Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses 2 -Fair. While the proposal broadly addresses the criterion, there are significant weaknesses 1 3 -Good. The proposal addresses the criterion well, although improvements would be necessary 2 4 -Very good. The proposal addresses the criterion very well, although certain improvements are still possible 2 5 -Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor 1 Problems which can’t be solved in grant agreement negotiations 2 Your comments must identify the required improvements

How to fill in IER - General Your comments Comments are confined only to the criterion concerned Comments describe only your final view of the proposal Comments are clear and unambiguous. Try to avoid obscure acronyms and technical terms Comments are of adequate length and cover all the bullet points under each criterion Comments provide full justification for the score given

Comments are substantial; do not write generic criticisms; be specific, explain Comments are facts not opinions, don’t show doubt or indecision not “I don’t understand why....” but “The proposers do not make clear why...” Poor comments include words like: “Perhaps, think, seems, assume, probably, …” Good comments include words like: “Because, percent, specifically, for example, …” IER – Be factual

Poor comments are vague - Good ones are precise “I think the consortium management plan is probably inadequate.” “The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.” “The resources for the project seem unrealistic.” “The resources in Workpackages 4 and 6 are seriously underestimated given the complexity of the activities involved.” IER – Give Clear Messages

IER – Avoiding a dispute Poor comments provide an opening for a debate - Good comments close the issue “There is no discussion of dissemination activities.” “Dissemination activities are not adequately discussed.” “There is only one end-user organisation in the consortium.” “The consortium lacks a sufficient participation of end- users.” “The proposal coordinator is not adequately experienced.” “The proposal coordinator does not demonstrate in this proposal an adequate level of experience of work in this field.”

IER – Varying the Vocabulary Why say “Poor” when you can say: Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile… Why say “Excellent” when you can say: Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully- prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …

IER – Final Check Have you fully explained the proposal’s strengths and weaknesses on each of the criteria ? Do your scores match your comments (high scores = positive comments, low scores = negative comments)? Have you double-checked any matters-of-fact which you have quoted? Have you written at adequate length? If this was your proposal, would you find this report fair, accurate, clear and complete?

ICT Evaluations Submitting your IERs

First, read over all your proposals to get an idea of the general standard and content When you have completed the evaluation of each proposal, submit your IER (IAR) for that proposal in the Rivet tool - Don’t wait until you have evaluated all your proposals before starting to submit them You will see that the Rivet tool has the facility to create Consensus Reports CRs and Evaluation Summary Reports ESRs. We will not use this in this evaluation; you stop at the creation of IERs (IARs)

Submitting your IERs Please respect the deadline you have been given for completing and submitting your forms Don’t worry if, after submitting your IER, you would like to add to or modify your comments. You will anyway get a chance to discuss your opinions with the other experts in the meetings in Brussels Bring your copies of the proposals and a copy of your IERs with you to Brussels. In the meantime, keep them safe!

Give a fair and clear opinion on each proposal. You are: Independent : you represent yourself, not your employer, not your country…..) Objective : you evaluate the proposal as written Accurate : you use the official evaluation criteria only Consistent : you apply the same standard of judgment to each proposal Incommunicado : you do not disclose to anybody the contents of the proposals which you see Finally

Thank you very much for your help! - The ICT evaluation team