1. Study on the monitoring and evaluation systems of the ESF [Contract VC/2017/0131]
4 online surveys + selection of 53 EPs based on several criteria: all MS covered, category of region, multifund/monofund, YEI, use of CIE and budget.
Results presented based on replies and clarifications received. New replies and clarifications helped in developing proposals and difficulties, but no changes in the characterisation of the systems.
Replies from all MS for Survey A & B. Survey C, no replies from: BE, DE, ES, FR, IT, MT, RO. Survey D, no replies from: ES, IT, MT. Consultation, no replies from : DE, HR, HU, MT
We present preliminary findings, not focusing on statistics, but rather on interpretation and analysis of problems and solutions. Not all problems are presented here, only most important ones. Report includes analysis of all issues identified. Further assessment complemented with other sources is foreseen.
ESF Evaluation Partnership Meeting
7th December 2017, Brussels

2. Monitoring: problems and possible solutions
Preliminary results
Monitoring: problems and possible solutions
Monitoring requirements - Problems
Complexity of requirements
Relevance & specification of (common) indicators
Possible solutions
Reduce number of indicators
Introduce flexible reporting requirements
Revise indicators:
To harmonise indicators/definitions across funds
To facilitate use of administrative registers
To broaden scope (e.g. education and soft indicators)
To simplify complex indicators (e.g. other disadvantaged)
Problems:
Complexity of requirements: MAs: understanding (67%) and coordinating with beneficiaries (65%). MCs: monitoring and reporting requirements challenging, regulations too detailed. difficult to understand, coordination between MAs and beneficiaries difficult, regulations too detailed.
Relevance and specification of common indicators: indicators focused on labour market status and transitions not appropriate for all priority axes
Solutions:
Reduce number of indicators: Different views (PS vs common indicators). MA & beneficiaries would like to see the personal data collection requirements reduced (removed or made optional). Particular mention was made of sensitive personal data - not only to reduce the administrative burden, but also to avoid the risk of deterring participation (see below in relation to primary data collection) – and indicators related to household situation.
Flexible reporting requirements: report only on indicators relevant to the specific activities, outputs and results of a programme (e.g. by TO).
Harmonise indicators across funds: particularly beneficial (in terms of reducing the burden) in case of existing multi-fund monitoring systems and could help to pave the way for more such systems in future. MC respondents also stressed that this would make the job of evaluation and monitoring easier.
Facilitating use of registers: by having more flexible definitions of the indicators, for example (see section on use of registers).
Implies dropping indicators for which information cannot be obtained from registers (e.g. homelessness, improved labour market situation, household situation, etc.).
Limits the potential use of soft indicators
Study on the monitoring and evaluation systems of the ESF

3. Monitoring: problems and possible solutions
Preliminary results
Monitoring: problems and possible solutions
Data collection (direct data collection) - Problems
High volume of data requested
Difficulties with data collection on entry
Difficulties with data collection upon and after exit
Possible solutions
Reduce the number of variables (part. sensitive data and hh situation)
Possible solutions (MS level)
Increase the use of existing administrative data
Good practices reported by beneficiaries:
allocation of staff to assist participants with questionnaires
simplification of questionnaires (easy to understand language)
collecting data during implementation instead upon entry
Problems:
Volume of data requested: Most common problem for beneficiaries and MAs
Collecting data on entry: Collection of personal data: Participants refusing to provide personal data, obtaining data for minors, strict data protection laws. Data collection methods/forms: e.g. questionnaires too difficult, problems for migrants, difficulties for those with limited IT skills to fill online applications, etc.
Collecting data upon and after exit: Collecting data for participants that dropped out (Time of exit not known in advance, thus, data collection cannot be scheduled.), Reaching specific groups (e.g. migrants and youth that change contact details), Low response rates from participants 6 months after exiting the operation.
Study on the monitoring and evaluation systems of the ESF

4. Monitoring: problems and possible solutions
Data collection (admin. registers) - Problems
Inconsistencies between variables
Tight privacy/data protection laws
Synchronising existing data with ESF observation points
Lack of cooperation of relevant organisations in the country
Possible solutions
Introduce more flexible indicator definitions
Tolerate differences between available data and ESF observation dates
Total OPs using administrative data: 16/68 OPs covered by the survey (13 MS)
Problems:
- Inconsistency between variables: administrative registers do not cover all variables and/or population; differences in definitions
>Problem with participants refusing to provide personal data required to link registers (e.g. ID) may also occur.
Possible solutions (MS level)
Take necessary legal steps to facilitate/increase use of existing admin. data

5. Monitoring: problems and possible solutions
Preliminary results
Monitoring: problems and possible solutions
Use of data – Problems
MC meetings: difficulties with the assessment of performance/progress
Impact evaluations: limitations in available data
Performance incentives: issues to define relevant targets and baselines, unintended outcomes from performance incentive system
Possible solutions
Improve access to monitoring and performance data (Open Data Portal delayed)
Organise training and mutual learning events for MAs and MCs in the preparation of OPs
Monitor costs per output or per result
Proposals on the relevance of indicators (mentioned above)
Problems
Two fifths of MC respondents (39%) reported some difficulties with the assessment of performance and progress, though fewer (23%) had issues with access to information to review the OP. MC meetings: Recurring issues: timeliness of information and inappropriate presentation of data, meetings organised at short notice, with insufficient time for preparation, and not sufficient time allocated for the discussion of performance and effectiveness.
Unintended outcomes: use of targets to limit funding, discouragement of potential beneficiaries, encouraging rapid use of funds ahead of result orientation, “creaming” of participants.
Possible solutions:
Training and mutual learning events to set up targets and baselines.
Seven in ten MCs (72%) supported the idea of monitoring costs per output or per result. Concerns about admin. burden, comparison of results of different types of intervention, methods required complicated and not cost efficient.
Other possible solutions mentioned:
- 50% of MCs welcome more active involvement of the Commission in their meetings (i.e. not only as an observer). Rather than having voting rights, the respondents suggest that the EC should have an increased competence in making recommendations, bringing good examples from other countries, raising new ideas and drawing attention to recent developments.
Proposal from MCs was to have a common operational framework/guidelines for MCs- adopting a common approach to assessment processes could improve efficiency.
Three quarters (76%) of MC respondents were in favour of a more efficient system of performance incentives. But no specific proposals were made in this respect.
Possible solutions (MS level)
Improve access to monitoring and performance data (e.g. IT, NL & LT)
Study on the monitoring and evaluation systems of the ESF

6. Evaluation: problems and possible solutions
Preliminary results
Evaluation: problems and possible solutions
Quality of evaluation plans - Problems
Evaluation questions not always specified
Evaluation methods confused with data collection methods and not linked to envisaged questions
Little information on data requirements & data collection procedures and on use of findings
Possible solutions
Common obligatory evaluation questions (MCs in favour, MAs mixed views)
Require detailed information on data requirements in EvPs
(77%) reported no difficulty with preparing EvP, however, weaknesses were identified in assessments of EvP by Helpdesk.
Some of these shortcomings may reflect a deliberate intention on the part of MAs to allow expert contractors to propose the most appropriate evaluation methodology and questions. However, this strategy has various drawbacks, in particular that the data required for the methods proposed may not be available when the evaluation is undertaken, which can result in significant delays and/or compromises in the quality and robustness of the evaluations.
EC competence:
Common evaluation questions:
In favour: simplify and accelerate the evaluation process (for instance by helping MAs in the preparation of the evaluation plans and in drafting the tender specifications), and facilitate the comparison and aggregation of results within and across Member States. Questions should be very general and basic, separate for each TO/IP.
Against: difficult to relevant and useful for all ESF-programmes; data availability and the needs of the supported groups; extra administrative burden.
Respondents were also asked about mid-term evaluations: (MAs against, MCs in favour)
Favour: would help to make necessary amendments to the OPs.
Against: few results available; particularly when operations do not begin at the same time and have varying speeds of implementation; difficult (and costly) to conduct multiple simultaneous evaluations
MA competence:
- Partnership with data providers: in order to better plan data availability, data needs and data collection.
- Common evaluation framework: help rationalise evaluations and ensure coherence. Common guidelines for all OPs can enable the harmonisation of practices across OPs, the active involvement of all OPs and, overall, faster and more efficient processes. It would give a minimum standard for all evaluations.
>Link to proposals on use of findings later (last slide)
Possible solutions (MS level)
Partnership with data providers to prepare evaluation activities (e.g. BG)
Common evaluation framework for all OPs in the MS (e.g. ES, PT)
Study on the monitoring and evaluation systems of the ESF

7. Evaluation: problems and possible solutions
Preliminary results
Evaluation: problems and possible solutions
Implementation of evaluations - Problems
Reasons for delay in evaluations:
Delay in implementation of OP /late start of projects
Issues with data availability
Difficulties with the procurement process (e.g. low quality offers)
Difficulties with the set-up of the evaluation system
Lack of resources within MAs
Possible solutions
Simplify evaluation requirements for small programmes
- 36 EvP: 615 evaluations, of which 90 evaluations already delayed.
Problems:
- Procurement process: e.g. low quality offers, may imply have to relaunch the process.
- Difficulties with evaluation systems: for example, governance, defining evaluation competences.
Solutions:
- Focus evaluation efforts in areas receiving majority of funds
- Terms of reference: data requirements (next slide on data availability) and ensure quality offers & relevant expertise (see later). ToR need to be comprehensive It is a key issue to address many problems in evaluations.
Possible solutions (MS level)
Include data requirements in the ToR and focus on quality criteria
Study on the monitoring and evaluation systems of the ESF

8. Evaluation: problems and possible solutions
Preliminary results
Evaluation: problems and possible solutions
Availability of data
Data required not available from MIS
Direct data collection difficulties
Difficulties in accessing administrative registers (legal restrictions)
Difficulties for data on control group
Possible solutions
Obligatory specification of data requirements in EvPs
Problems: Limited availability of appropriate/relevant data (63% of MAs)
Data not available from MIS: parallel data collection required.
Direct data collection difficulties: low response rates & personal sensitive information (similar issues mentioned in monitoring)
Difficulties for data on control group data: maybe related with both identifying control group (similar characteristics than ESF participants), as well as obtaining the required data from individuals.
Note: some problems on data availability are similar to those outlined in monitoring (data collection, use of registers). Solutions here only address the evaluation element.
>Interestingly, these issues arise not only in cases where data are collected directly from participants but also in cases where administrative registers are used, highlighting that even when registers are made accessible for the purposes of ESF monitoring/evaluation, the available data may not fulfil the evaluation needs.
>Difficulties in this respect reflect the fact that data needs and requirements have generally not been adequately dealt with in the evaluation plans so that no provision has been made to ensure that the necessary data would be available. Thus, the solution at EC level is:
Solutions – EC level:
Obligatory specification of data requirements in EvPs: so that data needs are taken into account in programme design
Solutions – MS level:
- The ToR, in addition to the data that are already available (e.g. from the MIS), should include what data needs to be collected.
Possible solutions (MS level)
Better definition of evaluation questions in EvPs
Include data requirements specifications in ToR for potential evaluators
Facilitate access to administrative data
Study on the monitoring and evaluation systems of the ESF

9. Evaluation: problems and possible solutions
Preliminary results
Evaluation: problems and possible solutions
Relevant expertise
Lack of technical expertise within MAs & MCs on evaluations, part. CIEs
Contractors without necessary expertise due to selection criteria
Difficulties in finding suitable experts (ESF & evaluation techniques)
Possible solutions
Facilitate access to evaluation expertise and good practices
Possible solutions (MS level)
Strengthen capacity of MAs (e.g. training, mutual learning events)
Reduce the weight of price in the award criteria
Increase the maximum price in the ToR to attract evaluators
Encourage involvement of academic evaluation experts
Problems:
Difficulties in expertise in CIE: partly because this method was not used in the previous programming periods)
Contractors without necessary expertise due to selection criteria: too much emphasis on price over technical quality.
Difficulty to find suitable experts exacerbated by the increased number of evaluations required and is more problematic in small countries.
Solutions are not related to evaluation requirements at EU level.
Increase maximum prices: For 80% of the MA respondents, the financial resources available for evaluations are not an issue. Higher prices may also attract foreign experts with more suitable profiles.
Study on the monitoring and evaluation systems of the ESF

10. Evaluation: problems and possible solutions
Preliminary results
Evaluation: problems and possible solutions
Use of findings
Evaluations from previous programming period completed after OP design
Evaluations do not assess impact (e.g. too general)
Limited information and lack of formal procedures to ensure follow-up of recommendations
Possible solutions
Include section in all AIR to report on follow-up of recommendations
Problems:
Evaluations completed after OP design: see possible solutions on implementation of evaluations
Evaluations do not assess impact: see possible solutions on relevant expertise/quality of evaluation plans. Sign of being just administrative procedure.
Limited information on the follow-up of recommendations in the AIR (assessment of submitted AIR), and lack of formal procedures.
Solutions:
- Section in AIR on follow-up of recommendations: this is already in AIR2017, AIR2019 and final report. Although information is lacking in many cases.
Most MAs respondents agreed on usefulness of existing section on evaluation findings in AIR, although some mentioned administrative burden.
Possible solutions (MS level)
Examples of actions to follow-up of recommendations:
biennial report on status of each recommendation (IT)
upload of recommendations in the MIS (CZ)
indicator on % of recommendations addressed (ES)
Study on the monitoring and evaluation systems of the ESF

11. M&E of Conditional Support – Steps taken so far
Literature review on M&E schemes for Conditional Support (CS)
Adaptation of monitoring scheme from DG DEVCO/DG NEAR to ESF (and other DG EMPL funds) related CS
Adaptation of Comprehensive Evaluation Framework from OECD/DAC to ESF related CS
Construction of a potential intervention logic for an ESF related CS support
Analysis of the practical implications for M&E of CS along the lines of the intervention logic

12. Next steps?
Focus groups with selected participants (survey respondents & experts) will be held (Jan 2018) to debate proposals for improving monitoring and evaluation requirements for the next programming period.
Set of proposals considering 2 options:
Option A: unchanged ESF scope and delivery mechanisms (but with some possible changes)
Option B: ESF into a single Human Capital fund
Design of structure and formulation of M&E framework for a CS program
Focus groups:
Selection of participants has been based on various criteria from respondents (experience, type of stakeholders, MS, organization of ESF in the country, type of monitoring and evaluation system etc.
Other participants also invited (e.g. experts, EC officials).
A set of proposals for changes will be prepared based on preliminary findings complemented with other sources of information. They will include possible solutions presented here plus others. More detailed assessment of feasibility and implications of changes in M&E requirements.
Conditional support: final formulation of M&E framework for CS following any inputs required and establishment of conditions for the CS framework.
Study on the monitoring and evaluation systems of the ESF

13. Thank you! esf@applica.be
Study on the monitoring and evaluation systems of the ESF