1.
The Evaluation Phase
Juras Ulbikas

2.
Evaluation – Overview

3. Evaluation – Overview Selection of independent experts (=evaluators)
Evaluation – Overview
Selection of independent experts (=evaluators)
Submitted proposals
Eligibility check
Assessment (by evaluators)
Ranking
Report to coordinator

4. Evaluation – Overview Selection of independent experts (=evaluators)
Evaluation – Overview
Selection of independent experts (=evaluators)
From database
Balance of gender, specialism, countries, industry vs academic, etc…
Submitted proposals
Eligibility check
Pre-screening of data
Call objective
Assignment to evaluators

5. Evaluation – Who are the evaluators?
Evaluation – Who are the evaluators?
Highly-educated people
Specialist vs. generalist
Technically competent
Impact issues
Knowledge of English
Professional integrity
Evaluations on a personal basis
Professional, independent, impartial, and objective

6. Evaluation – Who are the evaluators?
Evaluation – Who are the evaluators?
Contract between evaluator and Commission
Conflict of interest
Confidentiality
An expert is human!
Novelty
Feasibility of the concept
Logical flow of arguments
European dimension
Consistency

7. Evaluation – Assessment
Evaluation – Assessment
First impression
Show that sufficient time and care has been spent on the proposal
Layout and structure
Tables, graphs. text margins, font size
Win or lose in the first two pages
The “instant kill”

8. Evaluation – Common evaluator annoyances
Evaluation – Common evaluator annoyances
Unnecessary repetition
Too much cross-referencing
Unclear intents of the proposers
Inconsistencies and no logical links
“Statements” and “conclusions” to be believed on the basis of the assumed authority of the applicants

9. Evaluation – Ratings Score Rating Explanation -
Evaluation – Ratings
Score
Rating
Explanation -
Missing or incomplete information 1
Very poor
The criterion is addressed unsatisfactorily 2
Poor
Serious inherent weaknesses 3
Fair
Significant weaknesses that would need correction 4
Good
The proposal addresses the criterion well, but improvements are possible 5
Excellent
The proposal successfully addresses all relevant aspects

10. Evaluation – Ratings Gender issues and ethical issues
Evaluation – Ratings
Gender issues and ethical issues
They do count but not formally rated
Weights may be applied
At the moment seldom applied

11.
Evaluation – Ranking

12. Evaluation – Criteria Evaluation criterion Threshold/ maximum score
Evaluation – Criteria
Evaluation criterion
Threshold/
maximum score
S&T quality
4/5
Impact
3/5
Implementation
Overall threshold
10-12/15

13. Evaluation – S&T Quality
Evaluation – S&T Quality
Soundness of concept and quality of objectives
matches the Work Programme?
clear objectives?
Progress beyond the state of the art
State of the art clearly described?
Quality and effectiveness of the S&T methodology and associated work plan
Sound and logical S&T approach, with clear technical route?

14. Evaluation – Implementation
Evaluation – Implementation
Management structures and procedures
Project management demonstrably of high quality?
Quality of the individual participants
Well-suited and committed participants ?
Quality of the consortium as a whole
Complementarity between consortium participants?
Allocation and justification of the resources to be committed (budget, staff, equipment)

15.
Evaluation – Impact
Contribution to the expected impacts listed in the Work Programme
Impact on reinforcing competitiveness or on solving societal problems
Clear added value in carrying out the work at European level
Account of research activities at national and European level initiatives.
Dissemination and/or exploitation of project results, and management of IPR

16. Evaluation Summary Report – ESR (S&T Quality)
Evaluation Summary Report – ESR (S&T Quality)
From a “Research for the benefit of SMEs” project: 4.5
[…] The state of the art is clearly described, systematic and provides a good assessment of flaws in current soft heeled shoes demonstrating a high level of scientific and technological innovation supported by a patent. As a result, the contribution to advancement of knowledge / technological progress is clear and based on a patented concept. […]
[…] There is a good risk analysis with appropriate control through milestones having go/no-go decision at M12. […]

17. Evaluation Summary Report – ESR (S&T Quality)
Evaluation Summary Report – ESR (S&T Quality)
From a “Research for the benefit of SMEs” project: 3.5
[…] The innovation level is not very high but it is a step forward in building technology which can be very useful if broadly used.[…]
[…] the focus of the project is still too much on basic research and not enough on its concrete application (e.g. no prototype is foreseen).[…]

18. Evaluation Summary Report – ESR (S&T Quality)
Evaluation Summary Report – ESR (S&T Quality)
From an NMP CP-IP project: 3.0
[…] Unfortunately, no details are given to be able to appreciate the progress beyond state-of-the-art regarding the weaknesses of existing technologies. The important energy consumption aspect for the envisaged equipment has not been clearly addressed. […]
[…] The project has unclear targets regarding material systems to be selected and tested (see for example WP1 & 2) in specific applications for which real advantages are expected. […]
[…] The proposers should have conducted prior lab scale investigations on certain material combinations […]

19. Evaluation Summary Report – ESR (Implementation)
Evaluation Summary Report – ESR (Implementation)
From a “Research for the benefit of SMEs” project: 4.0
The quality of the management structure and procedures is excellent.
A great number of measures is considered for succesful implementation of the project and they are all very clearly presented and defined and focused on the needs of the project.
The conflict resolution measures are realistic and workable.
The role of the coordinator demonstrates clear leadership is properly defined and appropriate experience is demonstrated. There is a good vertical integration within the SME partnership and, although some are very small, their roles are appropriate and competences relevant.
Overall this project got 13.0, and made it.

20. Evaluation Summary Report – ESR (Implementation)
Evaluation Summary Report – ESR (Implementation)
From a “Research for the benefit of SMEs” project: 4.0
The project shows a good structure of the management with experienced partners that have already participated in other related projects.
The consortium is trans-national but the partners from one of the participant countries seem to have an over-weighted role.
Overall this project got 11.5, and didn’t make it.

21. Evaluation Summary Report – ESR (Implementation)
Evaluation Summary Report – ESR (Implementation)
From an NMP CP-IP project: 3.0
The profiles of individual participants and personnel resources have been well presented.
There is no demonstration of the management capabilities of the Coordinator.
Another notable weakness is the lack of any procedures for resolving conflict in the various committees.
Regarding exploitation, the proposal only presents general principles and does not enter into the specifics of the work programme.
Overall this project got 10.0, and didn’t make it.

22. Evaluation Summary Report – ESR (Impact)
Evaluation Summary Report – ESR (Impact)
From an NMP CP-IP project: 4.0
[…] A serious weakness of the proposal is the lack of any in-depth financial analysis for the growth of the industrial sectors related to processing and material development. In this respect, energy consumption for the running of the equipment which is a key issue in materials processing is not addressed. […]

23. Evaluation Summary Report – ESR (Impact)
Evaluation Summary Report – ESR (Impact)
From a “Research for the benefit of SMEs” project: 4.0
[…] The project shows good exploitation and dissemination plans that covers academic society, industrial stakeholders and experts.
However, no demonstration activities are foreseen in order to show the real potentialities of the developed system.
The management of the IPR should be described in more detail.[…]