External Monitoring of the Fourth Call CAPACITIES Programme (e-Infrastructures) Brussels, 2 February 2009 Research Infrastructures Programme Committee Meeting Anna M. Assimakopoulos

Overall statement-Main Conclusions The observer is fully satisfied that the evaluation process was efficient, effective and overall valid. Evaluators and Commission officials acted fairly, equitably, ethically and with integrity. Consortia were given fair and equitable consideration, complying with standards laid down by the evaluation process. Consortia were given fair and equitable consideration, complying with standards laid down by the evaluation process. The evaluation process successfully identifies the best proposals. The evaluation process successfully identifies the best proposals. Proposals selected for funding are those which best meet the criteria and objectives detailed in the workprogramme. Proposals selected for funding are those which best meet the criteria and objectives detailed in the workprogramme. The remote evaluation has contributed to increased efficiency and produced added value in terms of quality of the results. The remote evaluation has contributed to increased efficiency and produced added value in terms of quality of the results.

Call Basics (1) 4th call for proposals under the “Capacities” Programme of the Seventh Framework Programme (e- Infrastructures) covering: INFRA : GÉANT; INFRA : Scientific Data Infrastructures (SDI) 4th call for proposals under the “Capacities” Programme of the Seventh Framework Programme (e- Infrastructures) covering: INFRA : GÉANT; INFRA : Scientific Data Infrastructures (SDI) The topic “Scientific Data Infrastructure” has a total available budget of 20 million €. It received 46 proposals, of which 44 were eligible, requesting funding of 146 million € which corresponded to an oversubscription of 7:1. The topic “Scientific Data Infrastructure” has a total available budget of 20 million €. It received 46 proposals, of which 44 were eligible, requesting funding of 146 million € which corresponded to an oversubscription of 7:1. 36 evaluators were engaged to assist in the evaluation of the written proposals, of which 9 were women; 9 came from the new Member States, and 2 from outside EU. 12 out of 36 came from the industry sector, with the remaining part from Universities, Research Centres and Consultancy firms. 4 experts were engaged as recorders. 36 evaluators were engaged to assist in the evaluation of the written proposals, of which 9 were women; 9 came from the new Member States, and 2 from outside EU. 12 out of 36 came from the industry sector, with the remaining part from Universities, Research Centres and Consultancy firms. 4 experts were engaged as recorders. 64% of the proposals submitted had SME participation. 11% of the participants in these proposals were SMEs and 10% of the requested funding was for SMEs. 64% of the proposals submitted had SME participation. 11% of the participants in these proposals were SMEs and 10% of the requested funding was for SMEs.

Call Basics (2) The topic GÉANT received 16 proposals, of which only one was eligible (15 were ineligible), requesting an overall funding of approximately 104 million €. The topic GÉANT received 16 proposals, of which only one was eligible (15 were ineligible), requesting an overall funding of approximately 104 million €. 6 evaluators were engaged to assist in the evaluation of the written proposals, of which none were women; 1 came from the new Member States, and 1 from outside EU. 5 out of 6 came from the industry sector, with the remaining part from Universities, Research Centres and Consultancy firms. 6 evaluators were engaged to assist in the evaluation of the written proposals, of which none were women; 1 came from the new Member States, and 1 from outside EU. 5 out of 6 came from the industry sector, with the remaining part from Universities, Research Centres and Consultancy firms.

11 Main Recommendations Preparative activities and logistics The planning of and preparation for the evaluation process carried out by the Commission officers was extremely well-organized, efficient and effective. Positive comments were made from all sides on the remote evaluation. The planning of and preparation for the evaluation process carried out by the Commission officers was extremely well-organized, efficient and effective. Positive comments were made from all sides on the remote evaluation. Recommendation No. 1: the observer considers that the remote reading was an overall success and recommends that it be adopted as a definitive practice. Recommendation No. 1: the observer considers that the remote reading was an overall success and recommends that it be adopted as a definitive practice.

11 Main Recommendations Selection of the Evaluators Recommendation No. 2: The observer reiterates the previous recommendation to continue and intensify efforts to attract women evaluators as well as evaluators from industry. Recommendation No. 2: The observer reiterates the previous recommendation to continue and intensify efforts to attract women evaluators as well as evaluators from industry.

11 Main Recommendations Evaluation Infrastructure and Security Recommendation No. 3: The observer reiterates the suggestion already made in previous Observation Reports that the Commission undertake an in-depth review of Pinocchio’s performance and its interoperability with other software systems used during the evaluation process (remote and on-site) Recommendation No. 3: The observer reiterates the suggestion already made in previous Observation Reports that the Commission undertake an in-depth review of Pinocchio’s performance and its interoperability with other software systems used during the evaluation process (remote and on-site) Recommendations regarding security checks made in the previous monitoring report were duly taken into consideration. Recommendations regarding security checks made in the previous monitoring report were duly taken into consideration.

11 Main Recommendations Evaluation Process-Briefings Recommendation No. 4: The observer underlines the significance of specific briefings of the dedicated recorders, especially those who are performing the task for the first time in order to further implement best practice, expedite the Consensus Meetings and harmonize and secure the high-quality of the Consensus Reports produced. Recommendation No. 4: The observer underlines the significance of specific briefings of the dedicated recorders, especially those who are performing the task for the first time in order to further implement best practice, expedite the Consensus Meetings and harmonize and secure the high-quality of the Consensus Reports produced. Recommendation No. 5: The observer also suggests advance distribution of a video of the general and specific briefings to be delivered to the experts during the remote phase of the evaluation as a supplement and complement to the material already distributed. Recommendation No. 5: The observer also suggests advance distribution of a video of the general and specific briefings to be delivered to the experts during the remote phase of the evaluation as a supplement and complement to the material already distributed.

11 Main Recommendations Evaluation Process-Individual Assessments-Marks Recommendation No. 6: The observer recommends that the Commission in continuing to carry out the individual assessment of proposals remotely, attempts to implement procedures to minimize the time experts spend on-site, simultaneously continuing to widely publicize the overall exercise to female experts and experts from industry with particular fervour. Recommendation No. 6: The observer recommends that the Commission in continuing to carry out the individual assessment of proposals remotely, attempts to implement procedures to minimize the time experts spend on-site, simultaneously continuing to widely publicize the overall exercise to female experts and experts from industry with particular fervour. Recommendation No. 7: As a “best practice” suggestion, it is recommended that the Commission staff steering the Consensus Meetings, frequently remind the evaluators of the precise meaning of the scores, by reading and/or projecting the definitions thereof. Recommendation No. 7: As a “best practice” suggestion, it is recommended that the Commission staff steering the Consensus Meetings, frequently remind the evaluators of the precise meaning of the scores, by reading and/or projecting the definitions thereof.

11 Main Recommendations Evaluation Process-Consensus Meetings Recommendation No. 8: As aforementioned, the overall evaluation exercise is highly “streamlined”. However, it is recommended to continually pursue pro-forma best-practice “process” for Consensus Meeting management. Recommendation No. 8: As aforementioned, the overall evaluation exercise is highly “streamlined”. However, it is recommended to continually pursue pro-forma best-practice “process” for Consensus Meeting management. Recommendation No. 9: It is recommended that these best- practice guidelines be presented also as part of the initial briefings and that they are coupled with specific training for new Project Officers and Recorders to ensure that they are continually observed. Recommendation No. 9: It is recommended that these best- practice guidelines be presented also as part of the initial briefings and that they are coupled with specific training for new Project Officers and Recorders to ensure that they are continually observed.

11 Main Recommendations Evaluation Process-Panel Meetings Recommendation No. 10: It is recommended that the questions posed should be read aloud by the Commission Officer or the Rapporteur at the beginning of each hearing and that the questions appear on the slides presented by the Consortium. Recommendation No. 10: It is recommended that the questions posed should be read aloud by the Commission Officer or the Rapporteur at the beginning of each hearing and that the questions appear on the slides presented by the Consortium. Recommendation No. 11: The observer recommends that the Commission review the policy of inviting all proposals above threshold to the hearings. This review can include a statistical overview related to the number of proposals invited to hearings that are actually approved and the average score cut-off for final approval. Solutions such as an increased threshold, a hearing cut-off based on an oversubscription percentage of the available budget (e.g. 200%) or providing proposers with information about the chances of approval should be considered as part of this review. Recommendation No. 11: The observer recommends that the Commission review the policy of inviting all proposals above threshold to the hearings. This review can include a statistical overview related to the number of proposals invited to hearings that are actually approved and the average score cut-off for final approval. Solutions such as an increased threshold, a hearing cut-off based on an oversubscription percentage of the available budget (e.g. 200%) or providing proposers with information about the chances of approval should be considered as part of this review.

Conclusions (1) Conclusions (1) Overall, the observer finds that the evaluation process was carried out fairly, equitably and transparently. The rules and guidelines set out for the evaluation were carefully followed in all phases and aspects of the evaluation. Overall, the observer finds that the evaluation process was carried out fairly, equitably and transparently. The rules and guidelines set out for the evaluation were carefully followed in all phases and aspects of the evaluation. There is no doubt that everyone involved in the evaluation process, evaluators and Commission staff, have shown dedication to ensuring that all proposals and respective consortia are given due consideration according to the standards laid down in the evaluation process. There is no doubt that everyone involved in the evaluation process, evaluators and Commission staff, have shown dedication to ensuring that all proposals and respective consortia are given due consideration according to the standards laid down in the evaluation process. The observer is satisfied that the evaluation process as executed in this call has proved effective in ensuring that the proposals selected for funding are those which best fit the aims and objectives as detailed in the workprogramme. The observer is satisfied that the evaluation process as executed in this call has proved effective in ensuring that the proposals selected for funding are those which best fit the aims and objectives as detailed in the workprogramme. Conflicts of interest and resubmissions were handled in an appropriate way. Conflicts of interest and resubmissions were handled in an appropriate way.

Conclusions (2) Conclusions (2) The evaluation exercise itself, has reached a stage of maturity by which, in general, processes and procedures run smoothly. The evaluation exercise itself, has reached a stage of maturity by which, in general, processes and procedures run smoothly. The evaluation has now fully “acclimated” to the new facilities. IT infrastructure still poses challenges in terms of functionality and interoperability. The evaluation has now fully “acclimated” to the new facilities. IT infrastructure still poses challenges in terms of functionality and interoperability. The remote reading has been an overall success and made a positive contribution to the process, enhancing its efficiency, effectiveness and quality. The remote reading has been an overall success and made a positive contribution to the process, enhancing its efficiency, effectiveness and quality. Potential further efficiency gains, balanced against fairness and quality considerations should be examined in the context of reviewing the “above-threshold” invitation rule for hearings. Potential further efficiency gains, balanced against fairness and quality considerations should be examined in the context of reviewing the “above-threshold” invitation rule for hearings. Commission officers have exhibited a high level of professionalism and courtesy in creating a proactive yet strict working environment yielding high-quality and well-documented results, while remaining open to suggestions for further improvements. Commission officers have exhibited a high level of professionalism and courtesy in creating a proactive yet strict working environment yielding high-quality and well-documented results, while remaining open to suggestions for further improvements.