FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT 1 Joint Action on Health Workforce Planning and Forecasting M.Vallimies-Patomaki, J.Lammintakanen Brussels – April 12, 2013 Work Package 3 Finland Evaluation of the Joint Action

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Agenda of the WP3 kick-off meeting Introduction of WP3 partners Scope and objectives of WP3 Deliverables Preparation of the evaluation strategy Discussion topics 2

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Core team Dr Marjukka Vallimies-Patomäki, WP leader, Ministry of Social Affairs and Health, Finland Mr Jesmond Sharples, WP Co-leader, Ministry of Health, Malta Prof. Juha Kinnunen, Senior Management Scientist, University of Eastern Finland, Finland Prof. Johanna Lammintakanen, Management Scientist, University of Eastern Finland, Finland 3

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Partners DG SANCO D2/EU, EAHC/EU Ministry of Social Affairs and Health/FI, Ministry of Health/MT, Ministry of Public Health/BE, Ministry of Health/PL, Ministry of Health/SK University of Eastern Finland/FI, Medical University of Varna/BG, NCPHA/BG, University of Bremen/DE, Semmelweis University/HU CfWI/UK EFN, EHMA 4

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Scope and objectives  WP3 provides a systematic approach for the internal evaluation of the Joint Action  Aim is to verify if the JA is being implemented as planned and achieving the objectives: - output evaluation is to verify compliance to achievement of the objectives - outcome evaluation is to assess usefulness of outcomes, recommendations and sustainability beyond the Joint Action - process evaluation is to ensure compliance to the rules and implementation 5

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Engagement of the member states and stakeholders 1.Mentoring group: - WP leaders and representatives of stakeholders - Discussing and giving feedback of the evaluation strategy 2.Expert reference groups: - Representatives of member states and stakeholders - Discussing and appraising the quality of the deliverables 3.Focus groups: - Representatives of member states and stakeholders - A ppraising the impact and sustainability of the JA 6

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Deliverables Evaluation strategy and evaluation tools, August 2013 Expert reference group report, November 2014 Intermediate evaluation report, January 2015 Expert reference group report, December 2015 Final evaluation report, March

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Schedule for the preparation of the evaluation strategy 1 st draft by April 30/ Commission & WP1 Basic contents and evaluation indicators Comments by May 22 Evaluation indicators and spesific objectives: consultation through teleconference/Commission, WP1 &WP leaders 2 nd draft by June 7 /Commission, WP1 & WP leaders Comments by June 19(Dates changed by WP3) Stakeholder analysis, in collaboration with WP2 (> into an appendix) 3 rd version by June 30/ Commission, WP1, WP leaders, partners, strategic stakeholders Comments by July 31 Final version to WP1 by August to be submitted to EB 8

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Materials and methods 9 ApproachMaterialMethodEvaluation Tools Output evaluation Deliverables / WP2,4,5,6 and 7 Review process / Executive Board, MO Expert reference group Quality criteria for deliverables; semi- structured Outcome evaluation Changes in the HWF capacities / WP6 Deliverables / WP2 and 7 Content analysis / In collaboration with WP6 Delphi rounds/ Focus groups Criteria for comparing the Cprofiles to updated Cprofiles Interview themes Process evaluation Revisor´s reports / WP1 Progress reports / MO and WPs Content analysisMainly structured criteria for reports

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT 1. IntroductionThe aim of the evaluation (WP leader) Some words on the whole EUHWForce project Role, aims and tasks of evaluation in relation to JA 2. Evaluation Team Members and division of work (WP leader) Short introduction of members and areas of expertise Responsibilities 3. Evaluation reports and timeline Description of each evaluation deliverable Interim evaluation report Final evaluation report Data and methods used (J.Lammintakanen) Documents (stage plans), formal reports Mentoring group Focus groups Expert reference groups Timeline (in relation to GA) Report objectives 4. Ev. criteria and tools WP2, 4 - WP7, indicators (In collaboration with Commission, WP1, WP leaders, University of Bremen etc.) 5. Risk analysisE.g. variety of laguages, cultural manners, unrealialistic milestones, conflicting interests 10

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Review of spesific objectives Spesific Spesify the target group and the factors that need to be changed Measurable Formulate objectives in a measurable format Appropriate Make sure objectives are acchievable, attainable, acceptable for the target group Realistic Ensure that you can realisticly acchieve the objectives given the project resources Timebound Connect objectives to a timeline 11

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Key performance indicators Ensure that WP leaders are involved in identifying KPIs for all WPs of the JA KPI help the JA define and measure progress toward desired outcomes KPIs are quantifiable (measurable) measurements, agreed to beforehand, that reflect the critical success factors of the JA School: graduation rates of the students Social service organization: the number of clients assisted during the year Source: 12

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Overview of the discussion topics Evaluation tools: -Find examples and experiences Spesific objectives and evaluation indicators :  Review in collaboration with Commision, WP1 and WP leaders Stakeholder analysis:  In collaboration with WP2 Preparation of the evaluation strategy:  In collaboration with Commission, WP1, WP leaders, University of Bremen etc. 13

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Evaluation process: in co-operation with all WPs 14 Process evaluation (+ WP 1) Compliance to the rules Output evaluation (WP2,4,5,6,7) Compliance to the content Outcome evaluation (+WP2,6,7) Lessons learnt and recommendations ACHIEVEMENT OF THE JA OBJECTIVES, IMPLEMENTATION AND SUSTAINABILITY OF THE JA 2016 Each WP’s objectives Deliverable: Final report (> EB) 2013: SETTING OBJECTIVES FOR THE JA AND WPs ”… to create a platform for collaboration and exchange between MSs to better prepere for the future of the health workforce” Interim report > EB

FPS HEALTH, FOOD CHAIN SAFETY AND ENVIRONMENT Thank you for your attention ! 15