2013.10.01 The Evaluation Phase Juras Ulbikas

2013.10.01 Evaluation – Overview

Evaluation – Overview Selection of independent experts (=evaluators) 2013.10.01 Evaluation – Overview Selection of independent experts (=evaluators) Submitted proposals Eligibility check Assessment (by evaluators) Ranking Report to coordinator

Evaluation – Overview Selection of independent experts (=evaluators) 2013.10.01 Evaluation – Overview Selection of independent experts (=evaluators) From database Balance of gender, specialism, countries, industry vs academic, etc… Submitted proposals Eligibility check Pre-screening of data Call objective Assignment to evaluators

Evaluation – Who are the evaluators? 2013.10.01 Evaluation – Who are the evaluators? Highly-educated people Specialist vs. generalist Technically competent Impact issues Knowledge of English Professional integrity Evaluations on a personal basis Professional, independent, impartial, and objective

Evaluation – Who are the evaluators? 2013.10.01 Evaluation – Who are the evaluators? Contract between evaluator and Commission Conflict of interest Confidentiality An expert is human! Novelty Feasibility of the concept Logical flow of arguments European dimension Consistency

Evaluation – Assessment 2013.10.01 Evaluation – Assessment First impression Show that sufficient time and care has been spent on the proposal Layout and structure Tables, graphs. text margins, font size Win or lose in the first two pages The “instant kill”

Evaluation – Common evaluator annoyances 2013.10.01 Evaluation – Common evaluator annoyances Unnecessary repetition Too much cross-referencing Unclear intents of the proposers Inconsistencies and no logical links “Statements” and “conclusions” to be believed on the basis of the assumed authority of the applicants

Evaluation – Ratings Score Rating Explanation - 2013.10.01 Evaluation – Ratings Score Rating Explanation - Missing or incomplete information 1 Very poor The criterion is addressed unsatisfactorily 2 Poor Serious inherent weaknesses 3 Fair Significant weaknesses that would need correction 4 Good The proposal addresses the criterion well, but improvements are possible 5 Excellent The proposal successfully addresses all relevant aspects

Evaluation – Ratings Gender issues and ethical issues 2013.10.01 Evaluation – Ratings Gender issues and ethical issues They do count but not formally rated Weights may be applied At the moment seldom applied

2013.10.01 Evaluation – Ranking

Evaluation – Criteria Evaluation criterion Threshold/ maximum score 2013.10.01 Evaluation – Criteria Evaluation criterion Threshold/ maximum score S&T quality 4/5 Impact 3/5 Implementation Overall threshold 10-12/15

Evaluation – S&T Quality 2013.10.01 Evaluation – S&T Quality Soundness of concept and quality of objectives matches the Work Programme? clear objectives? Progress beyond the state of the art State of the art clearly described? Quality and effectiveness of the S&T methodology and associated work plan Sound and logical S&T approach, with clear technical route?

Evaluation – Implementation 2013.10.01 Evaluation – Implementation Management structures and procedures Project management demonstrably of high quality? Quality of the individual participants Well-suited and committed participants ? Quality of the consortium as a whole Complementarity between consortium participants? Allocation and justification of the resources to be committed (budget, staff, equipment)

2013.10.01 Evaluation – Impact Contribution to the expected impacts listed in the Work Programme Impact on reinforcing competitiveness or on solving societal problems Clear added value in carrying out the work at European level Account of research activities at national and European level initiatives. Dissemination and/or exploitation of project results, and management of IPR

Evaluation Summary Report – ESR (S&T Quality) 2013.10.01 Evaluation Summary Report – ESR (S&T Quality) From a “Research for the benefit of SMEs” project: 4.5 […] The state of the art is clearly described, systematic and provides a good assessment of flaws in current soft heeled shoes demonstrating a high level of scientific and technological innovation supported by a patent. As a result, the contribution to advancement of knowledge / technological progress is clear and based on a patented concept. […] […] There is a good risk analysis with appropriate control through milestones having go/no-go decision at M12. […]

Evaluation Summary Report – ESR (S&T Quality) 2013.10.01 Evaluation Summary Report – ESR (S&T Quality) From a “Research for the benefit of SMEs” project: 3.5 […] The innovation level is not very high but it is a step forward in building technology which can be very useful if broadly used.[…] […] the focus of the project is still too much on basic research and not enough on its concrete application (e.g. no prototype is foreseen).[…]

Evaluation Summary Report – ESR (S&T Quality) 2013.10.01 Evaluation Summary Report – ESR (S&T Quality) From an NMP CP-IP project: 3.0 […] Unfortunately, no details are given to be able to appreciate the progress beyond state-of-the-art regarding the weaknesses of existing technologies. The important energy consumption aspect for the envisaged equipment has not been clearly addressed. […] […] The project has unclear targets regarding material systems to be selected and tested (see for example WP1 & 2) in specific applications for which real advantages are expected. […] […] The proposers should have conducted prior lab scale investigations on certain material combinations […]

Evaluation Summary Report – ESR (Implementation) 2013.10.01 Evaluation Summary Report – ESR (Implementation) From a “Research for the benefit of SMEs” project: 4.0 The quality of the management structure and procedures is excellent. A great number of measures is considered for succesful implementation of the project and they are all very clearly presented and defined and focused on the needs of the project. The conflict resolution measures are realistic and workable. The role of the coordinator demonstrates clear leadership is properly defined and appropriate experience is demonstrated. There is a good vertical integration within the SME partnership and, although some are very small, their roles are appropriate and competences relevant. Overall this project got 13.0, and made it.

Evaluation Summary Report – ESR (Implementation) 2013.10.01 Evaluation Summary Report – ESR (Implementation) From a “Research for the benefit of SMEs” project: 4.0 The project shows a good structure of the management with experienced partners that have already participated in other related projects. The consortium is trans-national but the partners from one of the participant countries seem to have an over-weighted role. Overall this project got 11.5, and didn’t make it.

Evaluation Summary Report – ESR (Implementation) 2013.10.01 Evaluation Summary Report – ESR (Implementation) From an NMP CP-IP project: 3.0 The profiles of individual participants and personnel resources have been well presented. There is no demonstration of the management capabilities of the Coordinator. Another notable weakness is the lack of any procedures for resolving conflict in the various committees. Regarding exploitation, the proposal only presents general principles and does not enter into the specifics of the work programme. Overall this project got 10.0, and didn’t make it.

Evaluation Summary Report – ESR (Impact) 2013.10.01 Evaluation Summary Report – ESR (Impact) From an NMP CP-IP project: 4.0 […] A serious weakness of the proposal is the lack of any in-depth financial analysis for the growth of the industrial sectors related to processing and material development. In this respect, energy consumption for the running of the equipment which is a key issue in materials processing is not addressed. […]

Evaluation Summary Report – ESR (Impact) 2013.10.01 Evaluation Summary Report – ESR (Impact) From a “Research for the benefit of SMEs” project: 4.0 […] The project shows good exploitation and dissemination plans that covers academic society, industrial stakeholders and experts. However, no demonstration activities are foreseen in order to show the real potentialities of the developed system. The management of the IPR should be described in more detail.[…]