1. 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006

2. 2  Evaluation concepts  Objectives of the Futurreg On- going Evaluation  Theory Background  Evaluation Methodology  Evaluation activities

3. 3 The evaluation is a systematic and objective process which values the relevance, efficiency and effectiveness of the policy, programmes and projects in relation with the established objectives (Fahernkrog et al 2002). The evaluation has to carry out five levels of assessment: Relevance and Coherence Cost and Benefit, Efficiency Effectiveness Utility Sustainability.

4. 4 Internal Coherence Relevance Efficiency Effectiveness Utility / Sustainability Actual/Future Needs Objectives General/Spe cific Impact Results /Benefit Input/Cos t Activit y Outputs Relation of the evaluation process

5. 5 Why to Evaluate?  Control and Accountability – what have been done with all the money?  Knowledge about results – what are the effects of the activities and measures? Did we really achieve what we wanted to achieve?  Learning approach – what went well, which things went wrong, why did we (not) achieve the objectives? How can we improve working in the future?  Building link between involved actors

6. 6 ISSUES FOR FURTHER DEBATE Institutional Learning in project and programme management is necessary to exploit fully the potential of evaluations. Training and awareness-raising for both, project and programme managers and evaluators. Integrate Evaluation into the project-cycle, internalise the monitoring and evaluation function. Free evaluation from the stigma of being an instrument for control and punishment.

7. 7 Learning approach Taking desitions & policy makers building links Regional foresight capabilities Triangle of Futurreg evaluation functions Suggestion for improvement Implementation and impact Goal control and follow-up

8. 8  to provide information and recommendations in order to improve the Futurreg project and to contribute to the development of a regional foresight culture.  Evaluation in conformity with the provision under Interreg IIIC normative  To evaluate the relevance and coherence of Futurreg project.  To evaluate the utility and sustainability of Futurreg as a project.  To derive practical lessons and realistic recommendations which are important for the dinamization of foresight capabilities at regional level.

9. 9 The Foresight can be defined as a systematic, participatory, future intelligence gathering and medium-to-long-term vision-building process aimed at present day decisions and mobilising joint actions. (FOREN, EU, 2002)

10. 10  Complex Subject and Project, wide variety of stakeholders.  Since the project is still running, it is often not possible to identify or examine final results or impacts.  Limited visibility and sometimes indirect effects of Futurreg actions on final beneficiaries (SMEs, enterprises), leds to difficulties integrating their perspectives in the analysis to the same extent as the points of view of intermediaries.

11. 11 Main ObjectivesBasic sciencesKey TechnologiesInnovation Social Industrial Military defense Caracostas y Muldur (1998) 1945197020002020 Evolution of Innovation policy Futures& Foresight

12. 12 Policy makers’ environment Valovirta ja Hjelt (2005) Evolution and foresight: interrelations - Impact assesment - technology assesment

13. 13 Policy makers’ environment Valovirta ja Hjelt (2005) Evolution and foresight: interrelations

14. 14 Development of Evaluation Methodology Evaluation criteria for applications/implementation projects Development of indicators to evaluate each component of the project Development of a questionnaire for the partners On-going evaluation Questionnaire of satisfaction for the actual users of the application Interviews with the regional actors benefit of the project Ex-post evaluation report

15. 15 Methodological tools for evaluation: –Desk research (background literature, work programmes, project implementation reports, former foresight projects evaluations, studies, statistics, etc.) –Work research (Interviews with partnership and select stakeholders at regional level, On- line Questionnaire). –Permanent relationship with partner (meeting, e-mail, phone). Contrast dynamics and consent.

16. 16 Evaluation Framework of the project: Evaluation Topics Evaluation QuestionsIndicatorsData Sources Data gathering (Methods) MainAdditional Relevance Effectiveness Efficiency Effects and Impacts

17. 17 CONCLUSION Evaluation will give new insights on how to improve. Evaluation will give you information on which measures work and which don’t. Evaluation gives the public institutions a proof of what you have done and achieved, and legitimise what you do at regional level.

18. 18 The contribution of Futurreg Evaluation:  Ensures the RELEVANCE, FEASIBILITY and SUSTAINABILITY of a project  Facilitates a dialogue / OWNERSHIP  Ensures that fundamental questions are asked and weaknesses are analysed  Defines the key elements & the settings of a project  Identifies measurements/ indicators of the projects achievements  Systematic common sense (helping to adapt expectations to planned expenditure to time frames)

19. 19 Structure of the Report Executive Summary Introduction Evaluation Methodology Analysis of: –Futurreg deliveries –Futures toolkit –Regional foresight capabilities impact Conclusions and Recommendations on the Overall Futurreg