1. Study on the monitoring and evaluation systems of the ESF [Contract VC/2017/0131]
ESF Evaluation Partnership Meeting
11th April 2018, Brussels

2. Study on ESF Monitoring & Evaluation systems
Schedule
The final report is being discussed
To be finalised by end April and
shared with you
Objectives
Assess implementation of M&E systems
Identify challenges
Make proposals to improve M&E requirements for post-2020
M&E of ESF delivered under ESF conditional support
ESF used for the implementation of CSRs
Approach
Desk research and literature review
4 online surveys + consultation (53 EvPs) [Jun-Oct 2017]
Focus group discussions [Jan 2018]
Study on the monitoring and evaluation systems of the ESF

3. Recommendations of the study for post-2020: Monitoring
Data on participants:
“Multiple participation records” vs “single participation record”
Abandon problematic, irrelevant and complex indicators
Definitions for indicators accommodate national definitions (data registers)
NOT collected when evident
Breakdown by gender (women, men, non-binary gender)
Common output indicators:
Split into “core” and “non-core” indicators
Non-core: no need to fulfil completeness requirement, informed estimations
Presentation includes only main proposals for changes.
Multiple participation records: a different participation record for every time a participant enters the operation (one participant->multiple participation records). Provides comprehensive information on results for each participation. It allows the monitoring of individual costs. Does not pose incompatibilities with SCOs. Consistent with other EU data collections.
“Problematic indicators”: HH situation, other disadvantaged, from rural areas, migrants etc. can be kept as programme-specific indicators if relevant (e.g. rural areas used in RO, FR).
Definitions for indicators accommodate national definitions: e.g. unemployed
NOT collected when evident - e.g. for output indicators: "inactive" and "unemployed" for civil servants, the labour market status "employed" by default. For immediate result indicators: “in employment” for public employees, etc.
Study on the monitoring and evaluation systems of the ESF

4. Recommendations of the study for post-2020: Monitoring
List of common indicators on participants
Core output indicators
Non-core output indicators
All existing indicators on LM status, age and education and:
between 25 and 54 years of age
with lower secondary education or less (ISCED 0-2)
non-nationals,
- of whom from non-EU countries
with a foreign background or from minorities (including marginalised communities such as the Roma)
with disabilities
Immediate result indicators (upon leaving)
Longer-term result indicators
(6m after leaving)
in employment, including self-employment, upon leaving
engaged in job searching upon leaving
in education or training upon leaving
improved skills and competences upon leaving
- of whom gained a qualification
disadvantaged participants engaged in job searching, education/ training, gaining a qualification, or in employment, including self-employment
in employment, six months after leaving
- of whom in self-employment
in education or training
with an improved labour market situation, six months after leaving
participants above 54 years of age in employment, including self-employment
disadvantaged participants in employment, including self-employment
Main changes:
Remove above 54 NEETs, only inactive NEETs remains
Between (full age breakdown)
ISCED 0 included in educational attainment
Migrants – separated from previous indicator. Based on citizenship (as in previous PP). Common definition (comparability) and can be based on registers (less burden).
Indicator on foreign background& minorities will continue to be based on national definitions.
Study on the monitoring and evaluation systems of the ESF

5. Recommendations of the study for post-2020: Monitoring
List of common indicators on entities
Different approach: Counting “entities” vs “projects”  Result indicators
Output indicators
Number of supported public services
Number of supported public administrations
Number of supported enterprises
- of which SMEs (incl. cooperative enterprises, enterprises of the social economy)
Immediate result (upon leaving)
Number of entities with increased human capacity
Number of entities with improved tools and/or organisational structure
Number of entities with improved services
Number of entities with improved working conditions
Proposals based on review of existing PS indicators and desk research. Was not discussed in focus groups and surveys.
Number of entities with improved working conditions: to mention projects targeting women.
Study on the monitoring and evaluation systems of the ESF

6. Recommendations of the study for post-2020: Monitoring
Target setting
Cumulative targets for both common or programme specific indicators (no change proposed by the study) at the same level at which data are to be reported (no performance framework)
Targets expressed as percentages only if output indicators used as reference
Target setting and overall ESF expected achievement
Does it allows a calculation of the overall ESF expected achievement?
Is the system simple?
Is it neutral to the programme's intervention logic?
Does it accurately reflect OP expected achievements?
Target setting requirement for common indicators
Yes
(but not comprehensive) No
(lack of flexibility and loss of specificity of measures within OPs)
Quantification of contribution to common indicators
(administrative burden and risk of delaying OP adoption)
Estimation based on performance data and financial allocations by TO
Partially
(changes in financial allocations within the same TO are not reflected)
Target setting requirement for common indicators: set the obligation for MAs to set targets for at least one common output and one common result indicator.
Quantification of contribution to common indicators: it is required to identify for each programme-specific indicator one or more relevant common indicators and then to establish its contribution to the common indicator(s)
Estimation based on performance data and financial allocations by TO: to use monitoring data (achievement values, AV) from the current programming period and financial allocations (FA) of both the current and future periods to estimate the expected overall achievement, using the formulae below for each indicator, at the TO level
Study on the monitoring and evaluation systems of the ESF

7. Recommendations of the study for post-2020: Evaluation
Sound preparation of the evaluations
Requirements linked to ex-ante evaluations & EvPs (no change proposed by the study)
Methodological support: on CIE by CRIE (no change proposed by the study) + on TBE
Increase share of funds allocated to TA higher budget for evaluations
Strengthen mutual learning initiatives on evaluation techniques/practices
+ at the MS level:
Respect of partnership principle
Data access and availability
Clear legal basis in regulation to ensure access to admin. data for M&E purposes
Assess data needs & secure access to registers during the ex-ante phase
Strengthen cooperation with authorities managing registers
Clear indications in TOR of available data and data to be collected by the contractors
Early identification of actions to be evaluated using CIE approach and data needs
CIE: counter-factual impact evaluations
TBE: theory-based evaluations
TA: Technical assistance
COP: Community of Practice
Increase share of funds allocated to TA -> higher budget for evaluations (currently limited to 4% total amount of the Funds allocated to OPs under each category of region)
Larger number of evaluation competitors (incl. foreign experienced experts, and eventually, increased quality/more robust results.
Strengthen mutual learning initiatives on evaluation techniques/practices
Seminars, conferences, training, peer reviews
COP on CIE
Clear legal basis in regulation to ensure access to admin. data for M&E purposes -> for MAs and evaluators
Study on the monitoring and evaluation systems of the ESF

8. Recommendations of the study for post-2020: Evaluation
Evaluation capacity
Share examples of good practices between MS on how to prepare tender documents
+ at the MS level:
Reduce staff turnover within MAs (e.g. use of tops-up and bonuses co-financed by TA)
Wide dissemination of tenders (in EN) to address wider market of evaluators
TORs: limit language proficiency to qualitative aspects and weight of price (<50%)
Evaluation findings
Requirement to carry out impact evaluations (no change proposed by the study)
Publish a summary (in EN) of evaluation findings on CRIE archive page and introduce possibility to add a link to online summary in SFC
Competitions at EU level to increase visibility of robust evaluation findings
More structured template in AIR for reporting evaluation findings based on Better Regulation criteria
Learning events on the use of evaluation findings and follow-up of recommendations
NSI: National Statistical Institutes
Study on the monitoring and evaluation systems of the ESF

9. Monitoring CSRs implementation
Different types of country specific recommendations (CSRs)
Investment related CSRs (e.g. Take steps to increase labour market participation, in particular for women [...]’ (CSR Poland, 2017)
Systemic CSRs (e.g. ‘Reinforce the coordination between regional employment services, social services and employers, to better respond to jobseekers’ and employers’ needs’ (CSR Spain, 2017)
Different delivery modes possible
Reform delivery tool (RDT) focussing on systemic CSRs
Joint Action Plans and CSR Specific Priority Axis covering both CSR types
Result orientation and payment mechanisms
Fixed vs. variable tranche payments
Lump sum or cost based funding
Findings present the options, but many open issues remain
RDT: proposal put forward for post-2020 to support,
Complementary to current ESF
Implemented by direct management of the Members States
Funds will be disbursed (in full or in tranches) depending on the implementation and success of the reforms
Joint Action Plans (JAPs):
Grouping programmes/operations with common goals
Result oriented, i.e. payment-by-results
Specific Priority Axis (SPAs):
Similar to JAP but integrated into ESF framework as own priority axis
Fixed (security, no conditions)
Variable tranche payments (results/performance/implementation based).
Lump-sum: single payment made at a particular time (or broken down by tranches). (more suitable for structural/systematic)
Cost-based: following cost estimates (unit costs, flat rates).(e.g. SCOs). Suitable for investment CSR).
Study on the monitoring and evaluation systems of the ESF

10. Monitoring CSRs implementation
Application of a results-oriented approach
This requires:
Definition of appropriate result indicators (crucial for funding!)
Definition of both quantitative AND qualitative indicators
Strong and reliable monitoring system
Transparent and fair result assessment (across countries)
Clear rules how the size of payments/funding is related to the success of an operation (all-or-nothing or gradual approach)
EU semester
CSRs need to be clearly formulated for ESF implementation
All-or nothing vs gradual: will the funds be received only if 100% of targets achieved, or can they be funded if part of target achieved?
Study on the monitoring and evaluation systems of the ESF

11. Monitoring CSRs implementation - Uncertainties
How will the European Semester (CSRs) and the ESIF be harmonised in terms of timing?
Will the RDT cover systemic CSRs?
Shall the ESF cover only investment related CSRs or also systemic CSRs?
How can an overlap between CSR and ‘traditional’ ESF operations be avoided?
Are the monitoring systems prepared to define appropriate result indicators, and can they be collected in a reliable and stable way?
Once these issues are clarified, the monitoring mechanisms can be better specified.
Uncertainties – political discussions currently on-going, which will have an impact on tools for result-orientation and monitoring & evaluation requirements.
>European Semester (CSRs) – annual, vs ESIF multi-annual
>RDT scope (CSR systemic) and ESF link
>Type of CSRs covered by ESF (systemic an/or investment)
>Overlap: traditional ESF operations may already be covering areas of the CSRs (particularly investment ones). How to distinguish?
Details include :tranche vs. fixed payments, method of result assessment etc.

12. Thank you esf@applica.be
Open issues for discussion
Simplification of common indicators (participants & entities)
Data collection methods (registers & informed estimations)
Target setting methods
Other
Thank you
Study on the monitoring and evaluation systems of the ESF