1. European Cohesion Policy – European Social Fund Monitoring and Evaluation in 2014-2020
Effie Meletiou
Impact Assessment and Evaluation
DG Employment, Social Affairs and Inclusion
Nicosia, 26 November 2013

2. Outline Monitoring: highlights of regulatory requirements
Monitoring: regulatory requirements on indicators
Monitoring: programme specific indicators
Evaluation

3. Monitoring: Highlights of regulatory requirements (with comments)

4. Monitoring committee (Art. 41)
Set up within 3 months of adopting the OP
Single MC can cover more than one programme
MC draws up its own rules of procedure with reference to institutional, legal and financial framework of the MS concerned

5. Composition of MC (Art. 42)
Decided by MS provided that it is composed of relevant MS authorities, IBs and partners referred to in Art. 5.
Representatives of these partners shall be delegated through transparent processes.
Each MC member may have a voting right
The list of members shall be published

6. Functions of the MC (Art. 43)
MC shall meet at least once a year
Review implementation of OP(s)
Shall be informed of progress of achieving targets and milestones in the performance framework and results of qualitative analyses
Shall examine all issues affecting performance of the programme, including the conclusions of the performance review
Shall be consulted on OP modifications and give opinion, if it considers it appropriate
May make observations regarding OP implementation and evaluations, including actions related to reduction of admin burden on beneficiaries
Cohesion Policy
Regional Policy 6

7. Functions of the MC (Art. 100)
Shall examine:
Issues affecting performance
Implementation of evaluation plan and follow-up given to evaluation findings,
Implementation to communication plan, JAPs & financial instruments
Progress in fulfilling applicable ex ante conditionalities, where not fulfilled at the time of submission of OP or PA
Shall examine and approve:
Evaluation plan (covering one or several OPs) and any modification thereof
Communication plan
Modification of OP
Cohesion Policy
Regional Policy 7

8. Important changes in Regulations, Implementing and Delegated Acts
Enhanced focus on results
Increased importance of monitoring and evaluation
Even stronger need for clear intervention logic
Close link with Europe 2020 strategy
Delegated act: Art. 290 TFEU
Non-legislative acts of general application to supplement or amend certain non-essential elements of the legislative act
The objectives, content, scope and duration of the delegation of power to be explicitly defined in the legislative acts.
Council and EP can object within a period set in the legislative act
Implementing act: Art. 291 TFEU:
Where uniform conditions for implementing legally binding Union acts are needed, those acts shall confer implementing powers on the Commission
Enhanced focus on results Is NOT just a slogan without content:
Effectiveness as a general principle of the policy (art. 4(9) CPR)
Focus turns from inputs to outputs and results and to demonstrating the ESF impact and added value
Monitoring and evaluation play a key role. Much more attention needs to be paid to processing and assessing data.
BUT: It is not about producing more data! It is about producing better data.
OPs must contribute to achieving the objectives of the Europe 2020 Strategy Drafting the strategy shall not start from a blank sheet of paper but shall be based on previous analytical work in the context of Europe 2020:
Relevant country-specific recommendations
National Reform Programme
European semester analyses
Regional needs analysis, where relevant

9. Clear intervention logic
Composed of the hierarchy of programme objectives, actions, expected outputs and results
Important for all phases: programme design, implementation, monitoring and evaluation
E.g. choice of programme-specific indicators, time-planning of evaluations
The intervention logic is at the heart of the programming process but not clearly defined in the regulations. There is guidance on it. Also mention CRIE later

10. Clear intervention logic
Challenges and needs
Country-specific recommendations
National Reform Programme
European semester analyses
Consistent translation into thematic objectives and investment priorities
Funding priorities
Linkage to Europe 2020 Strategy

11. Programme theory – Intervention logic

12. Programme architecture
Thematic objective
Priority axis
Investment priority 1
Investment priority 2
Thematic objectives and investment priorities are objectives, not lists of activities – any actions contributing to these objectives can be pursued
If a priority combines more than 2-3 thematic objectives it is likely that the intervention logic is skewed = actions are put before objectives
Specific objective 1
Specific objective 2
Specific objective 1
Specific objective 2

14. Programming (1) Identification of development needs
Selection of thematic objectives and investment priorities as set out in the CPR and Fund-specific rules
Definition of at least one specific objective per investment priority to target the latter appropriately in the specific national or regional context
Definition of result indicators linked to specific objectives
The identification of development needs and challenges to be addressed is the starting point of programming
This is followed by the …..

15. Programme architecture
Thematic objective
Priority axis
Investment priority 1
Investment priority 2
Thematic objectives and investment priorities are objectives, not lists of activities – any actions contributing to these objectives can be pursued
If a priority combines more than 2-3 thematic objectives it is likely that the intervention logic is skewed = actions are put before objectives
Specific objective 1
Specific objective 2
Specific objective 1
Specific objective 2

16. Specific objectives
Art. 2 CPR - Definitions "specific objective' means the aim to which an investment priority or Union priority contributes in a specific national or regional context through actions or measures undertaken within such a priority"

17. Importance of specific objectives
Art. 17 CPR – ex ante conditionalities 'applicable ex ante conditionality' means a concrete and precisely pre-defined critical factor, which is a prerequisite for and has a direct and genuine link to, and direct impact on, the effective and efficient achievement of a specific objective for an investment priority or a Union priority

18. Importance of specific objectives
Art. 87(2)(b) CPR – Operational Programme For each priority axis: "the investment priorities and corresponding specific objectives" "in order to strengthen the result-orientation of the programming, the expected results for the specific objectives, and the corresponding result indicators, with a baseline value and a target value, where appropriate quantified in accordance with the Fund-specific rules" "a description of the type and examples of actions to be supported under each investment priority and their expected contribution to the specific objectives"

19. Importance of specific objectives
Annex II – performance framework "Milestones are intermediate targets, directly linked to the achievement of the specific objective of a priority, where appropriate, expressing the intended progress towards the targets set for the end of the period" Milestones and targets shall be "consistent with the nature and character of the specific objectives of the priority"

20. Equal opportunities
Art. 7 Equal opportunities "The Member States and the Commission shall ensure that equality between men and women and the integration of gender perspective are taken into account and promoted throughout the preparation and implementation of programmes, in relation to monitoring, reporting and evaluation."

21. Programming (2)
Only after the development needs, objectives and the result sought have been clarified, should one consider the types of actions to be supported, choosing (a mix of actions) which best achieves the objectives defined
Output indicators should be logically linked to and reflect the types of actions planned. Outputs generated by actions should also contribute logically towards the results that one aims to achieve.

23. Programming (3) Completing the drafting of the intervention logic
Reflection and adjustment of intervention logic
Testing the intervention logic – ex ante evaluation
The programming process should allow for a reflection on and adjustment of the intervention logic proposed
This is also one of the purposes of the ex-ante evaluation
The outcome should be a programme which includes a consistent and coherent intervention logic
The authorities should also be open to change the intervention logic in mid-period, if circumstances change or if the actions implemented are not sufficiently effective

24. General remarks on indicators
Common and where relevant programme-specific
Financial, output and result
No impact indicators (difference with )
Reported annually and electronically by MA as structured data as part of the AIR, broken down by investment priority
Reported as annual data, not cumulatively (difference with )
Relate to partially or fully implemented operations (definition in Art. 2 draft CPR)
Main instrument for monitoring
Input indicators measure the resources used
Output indicators relate to activities. They measure the volume of the activities.
Result indicators measure the direct and immediate effects on the direct beneficiaries
Impact indicators measure the wider and longer term consequences of the programme

25. General remarks on indicators
Recommendation: monitoring data entered into system throughout the year by beneficiaries/ bodies in charge of entering monitoring data
Clear name, unequivocal and easy to understand definition, measurement unit (provided for common indicators)
Monitoring: a tool to measure progress or detect deviation from initial plans/objectives

26. Monitoring information system
The system must record and store data on individual participants in a way that permits the managing authorities to perform the tasks related to monitoring and evaluation in conformity with the requirements set out in Art. 49 and Annex XX CPR and Articles 5 and 15(iv)(4) and (6), Annex I and II of the ESF Regulation
Tasks include:
undertake impact evaluations
Be able to contact participants after they have left the support
draw a representative sample of participants

27. Micro-data
CPR (Art. 114(2)(d)) sets out a legal obligation for the managing authorities to establish a system that records and stores individual participant data in computerised form
Micro-data of participants should be collected and stored
Micro-data are observation data collected on an individual object, i.e. a participation record
Observation data (characteristics and results) collected by indicators
Micro data allow MS to create output/result statistics and to match different observation data
May be complemented by unique personal identifiers

28. Data protection
The data processing arrangements must be in line with the provisions of Data protection Directive 95/46, in particular Articles 7 and 8 thereof.
Indicators require the collection of two categories of data on individuals:
Personal data – indicator marked with *
Personal sensitive data – indicator marked with **

29. Increased importance of monitoring - Delegated and Implementation Acts
Model for the OP (IA)
Performance framework: arrangements to ensure a consistent approach for determining milestones and targets and for assessing the attainment of the milestones and targets (IA)
Performance framework: financial corrections criteria (DA)
Data to be recorded and stored in computerised form (DA)
Model for the annual and final implementation report (IA)
Model for the progress report (IA)
I. Delegated (DA)/Implementing act (IA)
II. Preparation DA/IA before/after entry into force Common Provisions Regulation
III. Some delegated and implementing acts relevant for monitoring and evaluation
Delegated act: Art. 290 TFEU
Non-legislative acts of general application to supplement or amend certain non-essential elements of the legislative act
The objectives, content, scope and duration of the delegation of power to be explicitly defined in the legislative acts.
Council and EP can object within a period set in the legislative act
Implementing act: Art. 291 TFEU:
Where uniform conditions for implementing legally binding Union acts are needed, those acts shall confer implementing powers on the Commission
Article 290
A legislative act may delegate to the Commission the power to adopt non-legislative acts of general application to supplement or amend certain non-essential elements of the legislative act. The objectives, content, scope and duration of the delegation of power shall be explicitly defined in the legislative acts. The essential elements of an area shall be reserved for the legislative act and accordingly shall not be the subject of a delegation of power.
2. Legislative acts shall explicitly lay down the conditions to which the delegation is subject; these conditions may be as follows:
the European Parliament or the Council may decide to revoke the delegation;
the delegated act may enter into force only if no objection has been expressed by the European Parliament or the Council within a period set by the legislative act.
For the purposes of (a) and (b), the European Parliament shall act by a majority of its component members, and the Council by a qualified majority.
3. The adjective ‘delegated’ shall be inserted in the title of delegated acts.
Article 291
1. Member States shall adopt all measures of national law necessary to implement legally binding Union acts.
2. Where uniform conditions for implementing legally binding Union acts are needed, those acts shall confer implementing powers on the Commission, or, in duly justified specific cases and in the cases provided for in Articles 24 and 26 of the Treaty on European Union, on the Council.
3. For the purposes of paragraph 2, the European Parliament and the Council, acting by means of regulations in accordance with the ordinary legislative procedure, shall lay down in advance the rules and general principles concerning mechanisms for control by Member States of the Commission's
exercise of implementing powers.
4. The word ‘implementing’ shall be inserted in the title of implementing acts.

30. 1. Model for the Operational Programme (IA)
Performance framework for the priority axis
Broken down by Fund
Broken down by category of region
Key implementation steps, financial indicators, output indicators and where appropriate result indicators
Indicators are set at IP level, but the indicators for the performance framework have to be aggregated Indicators and implementation steps must be representative for the priority axis
Milestones for 2018 and targets for 2023

31. 2. Performance framework (IA): arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment
Main elements of the Implementing Act:
Arrangements for documentation of the establishment of milestones and targets
Basic requirements for different types of indicators
Arrangements for determining milestones and targets
Arrangements for the verification of the attainment of milestones and targets

32. 2. Performance framework (IA): arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment Arrangements for documentation of the establishment of milestones and targets
Methodologies and criteria to select indicators for PF ensuring milestones and targets comply with criteria of Annex II, § 3 CPR
Use of data/evidence and calculation method to estimate the value of milestones and targets
Rationale for the selection of output indicators, including explanation on share of the financial allocation represented by the operations which will produce the outputs + method to calculate the share -must represent more than 50% of financial allocation to the priority
Info on how methodology to ensure consistency in the PF has been applied in accordance with the provisions of the Partnership Agreement
Rational of selection of result indicators and key implementation steps
ANNEX II, point 3
Milestones and targets shall be
– realistic, achievable, relevant, capturing essential information on the progress of a priority;
– consistent with the nature and character of the specific objectives of the priority;
– transparent, with objectively verifiable targets and the source data identified and, where possible, publicly available;
– verifiable, without imposing a disproportionate administrative burden;
– consistent across operational programmes, where appropriate.
a) data or evidence (e.g. unit costs, benchmarks, standard or past rate of
implementation, expert advice, conclusions of ex-ante evaluation) used to estimate
the value of milestones and targets and the calculation method;
b) rationale for selection of output indicators, including an explanation of the share
of financial allocation represented by operations, which will produce the outputs,
as well the method applied to calculate the share, which must exceed 50% of the
financial allocation to the priority. In the case of priorities concerning more than
one Fund (the ERDF, ESF and CF) or the YEI or more than one category of
region, the rationale for the selection of output indicators has to be available by
Fund or YEI and by category of region. The share represented by the operations
has to exceed 50% of the financial allocation by Fund or YEI and by category of
region;
c) information on how the methodology and mechanisms to ensure consistency in
the functioning of the performance framework have been applied in line with the
provisions of the Partnership Agreement.

33. 2. Performance framework (IA): arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment Requirements for indicators and key implementation steps
Milestones and targets to be set at level of the priority
In case of multi-fund/multi-category of regions priority axes: breakdown by Fund and by category of region
For financial indicators: M&T refer to total amount of eligible expenditure entered into accounting system of certifying authority…
For ESF output indicators M&T refer to achieved value for fully or partially implemented operations
Key implementation steps refer to an important stage in delivery of a priority, with verified completion, and expressed as number or percentage.
Result indicators used where appropriate, closely linked to interventions
In case of YEI implemented as part of a priority axis: YEI is to be considered as a separate Fund (no need for a breakdown by category of region).
A fully implemented operation is an operation, which has been completed, but for which not necessarily all the related payments have been made by the beneficiaries or the corresponding public has been paid to the beneficiaries.
A partially implemented operation is an operation, for which the implementation has started, but which has not yet been fully implemented.
The key implementation step shall refer to an important stage in the delivery of a priority, whose completion is verifiable and may be expressed by a number or percentage. A key implementation step shall be used as a milestone in a situation where no measurable output is expected by the end of 2018 and shall not be used as target.
A key implementation step may be used if the outputs by the end of 2018 are expected to be insignificant, in which case it may refer to the number of selected operations that are to deliver such outputs by the end of programming period.

34. 2. Performance framework (IA): arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment Arrangements for verification of the attainment of milestones and targets
To be assessed at priority axis level
In case of multi-fund/multi-category of regions priority axes: assessment by Fund and by category of region
Achievement of milestone/target: all indicators of the performance framework of the priority axis have attained at least 85% of milestone (2018) or target (2023) value
Serious failure to achieve M&T:
Max 2 indicators/priority axis: any of the output or financial indicators failed to attain at least 65% of the milestone/target value
More than 2 indicators/priority: at least 2 of the output or financial indicators failed to attain at least 65% of the milestone/target value

35. 3. Performance framework (DA): financial corrections criteria
Conditions for applying financial corrections at end of programming period on basis of final implementation report:
Serious failure to achieve targets liked to financial indicators or output indicators
Max 2 indicators/priority: any of the indicators failed to attain at least 65% of target value
More than 2 indicators/priority: at least 2 of the indicators failed to attain at least 65% of target value
Serious failure is due to clearly identified implementation weakness
Commission previously communicated to MA the clearly identified implementation weakness
MS failed to take necessary corrective action
No socio-economic or environmental factors, no significant changes in the economic or environmental conditions in a MS…seriously affecting implementation of priorities concerned

36. EC reporting (Art. 46bis)
Reporting by the EC and debate on the ESI Funds
Three types of report: summary report and strategic reports and annual progress reports
Summary report:
Summary report based on AIRs, including summary of evaluations
Starting in 2016, to Council, EP, ECOSOC, CdR
Strategic report:
In 2017 and 2019, based on progress reports
Annual Progress Reports:
Starting in 2018, every two years
To spring summit, ESI Funds contribution to Europe 2020
Cohesion Policy
Regional Policy 36

37. Implementation reports - timeline
April 2015
YEI report (Annex I and II indicators for YEI funding) (Art. 15(iv)(3) ESF Reg.)
May 2016
light AIR (Art. 44(1) and (2) CPR), Annex I and II indicators, special reporting on YEI evaluation findings, Art. 15(iv)(4) ESF Reg.
June 2017
strategic AIR (Art. 44(3) and 101 CPR), Annex I (including data on homeless and rural area) and Annex II ESF Reg.
May 2018
light AIR, Annex I and II
June 2019
Strategic AIR (Art. 44(4) and 101 CPR), Annex I, including longer-term indicators, special reporting on YEI evaluation findings (Art. 15(iv)(4) ESF Reg.)
May
Light AIR, Annex I and II
December 2024
Final report, including longer-term indicators of Annex I, special reporting on YEI evaluation findings (Art. 15(iv)(4) ESF Reg.)

38. Monitoring: Regulatory requirements on INDICATORS

39. List of common ESF and YEI indicators
Structure of Annex I and II
Annex I - common ESF indicators
(1) Common output indicators on participants
(2) Common output indicators on entities
(3) Common immediate result indicators
(4) Common longer-term result indicators
Annex II - Youth Employment Initiative indicators
(1) Immediate result indicators
(2) Longer-term result indicators
Each of them to be reported for each investment priority – no selection possible for reporting

40. Annex I
Common indicators

41. Output indicators on people
Employment status
unemployed, including long-term unemployed*
long-term unemployed*
inactive*
inactive, not in education or training*
employed, including self-employed*

42. Output indicators on people
Age
below 25 years*
above 54 years*
participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training *
Education
with primary (ISCED 1) or lower secondary education (ISCED 2)*
with upper secondary (ISCED 3) or post-secondary education (ISCED 4)*
with tertiary education (ISCED 5 to 8)*

43. Inactive, not in education or training
Participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training
Unemployed, including
long-term unemployed
Inactive, not in education
or training
Above 54 years

44. Output indicators on people
Disadvantaged participants
participants who live in jobless households*
participants who live in jobless households with dependent children*
participants who live in a single adult household with dependent children*
migrants, people with a foreign background, minorities (including marginalised communities such as the Roma)**
disabled**
other disadvantaged**

45. Participants who live in jobless households*
All household members either unemployed or inactive
Household – housekeeping/social unit:
• having common arrangements;
• sharing household expenses or daily needs;
• in a shared common residence.
-> one person living alone / group of people, not necessarily related
-> living at the same address
Excluded:
- Households composed solely of students.
- Collective / institutional households (hospitals, old people’s homes, residential homes, prisons, military barracks, religious institutions, boarding houses and workers’ hostels, etc.)

46. Participants who live in jobless households with dependent children *
Subindicator of
Participants who live in jobless households*
Dependent children
all children under 17 years of age and
persons between years of age who are economically dependent on their parents
-> Participant can be any household member (parent, dependent child, other household member)

47. Participants who live in a single adult household with dependent children*
NO subindicator to "jobless household", but
same definitions regarding
household
dependent children

48. Output indicators on people
Disadvantaged participants
homeless or affected by housing exclusion*
from rural areas*
The data on participants under the above two indicators are to be provided in the AIR as specified in Article 44(3) of Regulation (EU) No [...] CPR (i.e. in 2017). They are to be collected based on a representative sample of participants within each investment priority.
Data for participants about rural areas are to be collected at LAU 2 (local administrative unit, former NUTS 5).
Data to be submitted in
June 2017

49. Homeless or affected by housing exclusion
National definition or ETHOS (European Typology of Homelessness and Housing Exclusion) definition – classification of four living circumstances : 1. Rooflessness (living rough / emergency accommodation), 2. Houselessness (in accommodation for the homeless, in women's shelters, in accommodation for immigrants, people due to be released from institutions and people receiving long-term support due to homelessness), 3. Insecure accommodation (insecure tenancies, under threat of eviction or violence), 4. Inadequate housing (unfit housing, non-conventional dwellings) -> Reference "Confronting Homelessness in the European Union"

50. From rural areas
Rural areas are to be understood as thinly populated areas according to the Degree of urbanisation (DEGURBA category 3) classification.
Thinly-populated areas means that more than 50 % of the population lives in rural grid cells.
The data shall be collected at the Local Administrative Unit level of LAU 2 (local administration/communes).The DE-GURBA category 3 shall be established according to header "for reference year 2012".

51. From rural areas

52. LAU 2 AT Gemeinden FI Kunnat / Kommuner NL Gemeenten BE
Gemeenten/Communes FR
Communes PL
Gminy BG
Населени Места/Naseleni Mesta GR
Demotiko diamerisma /Koinotiko diamerisma PT
Freguesias HR
Municipalities HU
Települések RO
Comune + Municipii + Oraşe CY
Dimoi, koinotites IE
Electoral Districts SE
Kommuner CZ
Obce IT
Comuni SI
občine DE LT
Seniūnijos SK DK
Sogne LU UK
Wards (or parts thereof EE
Vald, linn LV
Pilsētas, novadi, pagasti ES
Municipios MT
Kunsilli

53. Output indicators on entities
number of projects fully or partially implemented by social partners or non-governmental organisations
number of projects dedicated to sustainable participation and progress of women in employment;
number of projects targeting public administrations or public services at national, regional or local level
number of supported micro, small and medium-sized enterprises (including cooperative enterprises, enterprises of the social economy)

54. Number of projects dedicated at sustainable participation and progress of women in employment
Project with the aim of increasing the sustainable participation and progress of women in employment, thus combating the feminisation of poverty, reducing gender-based segregation and combating gender stereotypes in the labour market and in education and training, promoting reconciliation of work and personal life for all and equal sharing of care responsibilities between men and women. ESF Regulation, Art. 7, Promotion of equality between men and women

55. Immediate result indicators
inactive participants engaged in job searching upon leaving*
participants in education/training upon leaving*
participants gaining a qualification upon leaving*
participants in employment, including self-employment, upon leaving*
disadvantaged participants engaged in job searching, education/ training, gaining a qualification, or in employment, including self-employment, upon leaving**

56. Immediate result indicators
Disadvantaged are:
participants who live in jobless households*
participants who live in jobless households with dependent children*
participants who live in a single adult household with dependent children*
migrants, people with a foreign background, minorities (including marginalised communities such as the Roma)**
disabled**
other disadvantaged**
homeless or housing exclusion
rural area

57. Disadvantaged participants in job searching, in education/training, gaining a qualification or in employment upon leaving
Different logic than other immediate result indicators
All disadvantaged groups to be reported together
All immediate results to be reported together

58. Longer-term result indicators
participants in employment, including self-employment, 6 months after leaving*
participants with an improved labour market situation 6 months after leaving*
participants above 54 years in employment, including self-employment, 6 months after leaving*
disadvantaged participants in employment, including self-employment, 6 months after leaving**
These data are to be provided in the annual implementation reports as specified in Article 44(4) of Regulation (EU) No [CPR] (i.e & 2023). They are to be collected based on a representative sample of participants within each investment priority.

61. Participants in employment, including self-employment,
6 months after leaving
Including self-employment No separate reporting of self-employed 6 months after leaving

62. Targets for common result indicators
An output indicator as reference is needed. OP/AIR templates and SFC2014 will give a drop-down menue. Only those combinations will be possible.

63. Annex II
YEI indicators

64. Reported annually, including the longer-term result indicators
Reported in addition to the common indicators above
First report on YEI implementation due in April 2015, i.e. one year earlier than for the rest of the ESF

65. YEI immediate result indicators (I)
unemployed participants who complete the YEI supported intervention*
unemployed participants who receive an offer of employment, continued education, apprenticeship or traineeship upon leaving*
unemployed participants who are in education/training, gaining a qualification, or in employment, including self-employment, upon leaving*

66. YEI immediate result indicators (II)
long-term unemployed participants who complete the YEI supported intervention*
long-term unemployed participants who receive an offer of employment, continued education, apprenticeship or traineeship upon leaving*
long-term unemployed participants who are in education/training, gaining a qualification, or in employment, including self-employment, upon leaving*

67. YEI immediate result indicators (III)
inactive participants not in education or training who complete the YEI supported intervention*
inactive participants not in education or training who receive an offer of employment, continued education, apprenticeship or traineeship upon leaving*
inactive participants not in education or training who are in education/training, gaining a qualification, or in employment upon leaving*

68. YEI immediate result indicators
3 target groups x 3 immediate results = 9 immediate result indicators

69. YEI immediate result indicators
Completion of intervention
Attendance according to schedule until the last day/last session of scheduled end.
-> No recording as immediate result
if irregular attendance/ drop out.

70. YEI immediate result indicators
Offer of employment, continued education, apprenticeship or traineeship upon leaving Voluntary conditional promise, Indication of offeror's willingness to enter into agreement under specific terms with the participant Acceptance result in binding agreement with legal commitment of both parties

71. YEI immediate result indicators
Continued education
Enrolment in
formal education or
training programmes leading to recognised vocational qualification.

72. YEI immediate result indicators
Apprenticeship
Training contract or formal agreement (occupation, duration, skills to be acquired, wage or allowance etc.)
Directly or via the education institution
Normally part of formal education and training at upper secondary level (ISCED 3)
Duration on average 3 years
Successful completion leads to nationally recognised qualification

73. YEI immediate result indicators
Traineeships
Limited period of work practice spent at business, public bodies or non-profit institutions
Last a few weeks to a few months
Usually not considered to constitute employment contracts 

74. YEI longer-term result indicators
participants in continued education, training programmes leading to a qualification, an apprenticeship or a traineeship 6 months after leaving*
participants in employment, including self-employment, 6 months after leaving*
participants in self-employment 6 months after leaving*
The data for longer-term result indicators are to be collected based on a representative sample of participants within each investment priority.

75. Priority axes covering more than one category of region
Can be justified where where identical objectives and actions are pursued across all regions
Re-percussions:
Co-financing rate and financial management by priority axis and by category of region
Breakdown of performance framework and output indicators (and in case of ESF, also result indicators)by category of region
NB: the development needs of different categories of regions can diverge. In this case, the objectives and the intervention logic will also be different multiple priority axes

76. Baselines and targets Baselines for result indicators with a target
Cumulative target values for 2023 (for output and result indicators), if n+3 rule adopted
Targets are quantified for all output indicators (absolute numbers) and common result indicators (absolute numbers or shares/rates) and quantified or qualitative for programme-specific result indicators
Consult background papers for ESF target setting

77. Targets for common result indicators - example
Investment priority:
“Equality between men and women and reconciliation between work and private life”
Specific objective:
Increase participation of low skilled inactive or unemployed people with care responsibilities in the labour market
Targets for common result indicators should be set in function of the data reported for common output indicators.

78. Result target is often expressed in %
The % requires a reference value, i.e. the relevant output data.
CI Result: Participants gaining a qualification upon leaving
Targets:
50% gaining a qualification upon leaving WRONG!
50% of whom? Of all participants? No!
50% of the low skilled participants (ISCED 1 and 2) gaining a qualification upon leaving

79. CI Result: Participants in employment, including self-employment, upon leaving
Targets
50% in employment, including self-employment, upon leaving WRONG!
50% of whom? Of all participants? No!
50% of the unemployed in employment, including self-employment, upon leaving
40% of the inactive in employment, including self-employment, upon leaving

80. Monitoring: Programme specific indicators

81. Principles for sound indicators
Integrated set of indicators consisting of CI and programme-specific indicators
With a limited set of targets. Not all indicators need a target. But the most important ones do.
Indicators should cover the main scope of a priority (i.e. target group, type of activity etc.)
Targets should also cover large parts of a priority
Indicators should be simple - sophisticated data require evaluation

82. Programme-specific indicators – recommendation
Use common indicators and their definitions when establishing programme-specific indicators – decrease of administrative burden

83. Programme-specific longer-term result indicators
Possibility to monitor longer-term results over a longer time span than 6 months after leaving
If the same population/sample as for longer-term result indicators is covered – information on sustainability of results

84. Enhanced monitoring of results
With access to database with micro-data on employment status, enhanced monitoring could be envisaged:
Employment / unemployment spell of participants can be monitored in a prolonged period of time after leaving project

85. Time series with employment results of participants
Participants in employment x months after leaving, N=1000
Number of participants
Common longer-term result indicator
Programme-specific longer-term result indicators
Number of months after participants left project

86. Example from Belgium (Flanders)
People that never fell back into unemployment
After 6 months: about 1000 never worked
about 500 never worked
Source: presentation "Impact evaluation of PES action for the unemployed in Flanders" by Benedict Wauters (ESF Agency Flanders) and Steven Groenez (KU Leuven HIVA) given at ESF Evaluation Partnership

87. Programme-specific result indicators
Need to be logically linked to outputs (directly supported participants or entities)
Global effects on a target group are to be assessed through evaluations
Data comes from monitoring or surveys, or from existing databases building on micro-data
=> Avoid indicators based on general statistics, such as (increase of) employment rate in a region (by x%)
YES: participants aged 20 or younger starting an apprenticeship
NO: share of young people aged 20 or younger in apprenticeship

88. Three approaches to set programme-specific indicators
1) By combining different common ESF indicators
2) By combining common ESF indicators with programme specific characteristics/ features
3) By setting new indicators focusing solely on programme specific characteristics/ features

89. 1) By combining different one-dimensional common ESF indicators
Advantage: Data have to be collected anyway. Less administrative burden linked to data collection
NB: Some common ESF indicators are based on such combination
Participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training*
Disadvantaged participants engaged in job searching, education/ training, gaining a qualification, or in employment, including self- employment, upon leaving**
Participants above 54 years in employment, including self- employment, 6 months after leaving*
Disadvantaged participants in employment, including self-employment, 6 months after leaving**

90. Common output indicator: Participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training*
Above 54 years
Inactive, not in education or training
Unemployed, including long-term unemployed

91. Common longer-term result indicator: Disadvantaged participants in employment, including self-employment, 6 months after leaving**
Migrants, people with a foreign background, minorities (incl. marginalised communities such as the Roma)
Other disadvantaged
Disabled
Participants in employment, including self-employment, 6 months after leaving

92. Programme Output Indicator: Young low skilled inactive participants
Inactive, not in education or training
With primary or lower secondary education
Below 25 years

93. Programme Immediate Result indicator: high skilled unemployed in employment upon leaving
Unemployed, including long-term unemployed
With tertiary education
Participants in employment upon leaving

94. Programme Immediate Result indicator: low skilled older workers gaining a qualification upon leaving
Employed, including self-employed
With primary or lower secondary education
Above 54 years
Participants gaining a qualification upon leaving

95. 2) By combining common ESF indicators with programme specific characteristics/ features Advantage: indicators can take into account specific aspects of the programme.

96. Output indicator: NEETS
Participants between 16 and 24 years
Inactive, not in education or training
Unemployed, including long-term unemployed

97. Output indicator: Disadvantaged pupils
Pupils below 16 years
Migrant and minorities
Other disadvantaged
Disabled

98. Output indicator: Young high-skilled participants establishing a start-up upon leaving
Below 25 years
Start-ups
With tertiary education
Participants in employment, including self-employment, upon leaving or

99. Result indicator: Disadvantaged pupils in education or training upon leaving
Pupils below 16 years
Migrant and minorities
Other disadvantaged
Disabled
In education or training upon leaving

100. Result indicator: NEETs in education or training upon leaving
Inactive, not in education or training
Participants between 16 and 24 years
Unemployed, including long-term unemployed
Participants in education/
training upon leaving

101. 3) By setting new indicators focusing solely on programme specific characteristics/ features

102. This example shows possible subdivision of a common indicator into programme-specific indicators.

103. Steps in designing programme-specific indicators
'Deconstruct' the programme with view to the following categories:
Target groups (people & entities)
Type of planned activities
Themes
Projects
Type of expected results
Prioritise the content of each category in order to then identify indicators on the most important aspects
Develop indicators e.g. by combining characteristics/ features from the various categories
Review intervention logic to ensure that the most important aspects are covered by indicators

104. Example Investment priority:
“Equality between men and women and reconciliation between work and private life”
Specific objective:
Increase participation of low skilled inactive or unemployed people with care responsibilities in the labour market

105. Expected type of results
Target groups
Type of activity
Expected type of results
Support for child care/care for dependent persons
Training
Self-/Employment
Qualification
Inactive not in education or training /unemployed/ LTU with care responsibilities with ISCED below 3

106. Common output I with target
Programme-specific output indicators
CI results with targets
Programme-specific result indicators
With ISCED 1 or 2
Participants in employment, including self- employment, 6 months after leaving
Inactive not in education or training with care responsibilities with ISCED below 3
Unemployed with care responsibilities with ISCED below 3
LTU with care responsibilities with ISCED below 3
Participants gaining a qualification upon leaving
Low skilled (below ISCED 3) inactive or unemployed participants in employment who received support in their care responsibilities
Participants in employment, including self- employment, upon leaving

107. LTU with care responsibilities with ISCED below 3
Inactive not in education or training with care responsibilities with ISCED below 3
LTU with care responsibilities with ISCED below 3
Unemployed with care responsibilities with ISCED below 3
Inactive, not in education or training
Unemployed
LTU
With ISCED 1 or 2
Below ISCED 1
Participants in employment, including self-employment, upon leaving
With care responisibilitities
Gaining a qualification

108. Evaluation

109. What is hopefully going on right now

110. Programme design Intervention logic Ex-ante evaluation Indicators
Interactive process
Iterative process
Indicators
On-going evaluation
Implementation
Monitoring
Ex-post evaluation

111. The ex-ante should check for a clear intervention logic
Challenges and needs
Country-specific recommendations
National Reform Programme
European semester analyses
Consistent translation into thematic objectives and investment priorities
Funding priorities
Linkage to Europe 2020 Strategy

112. The ex-ante should check for a clear intervention logic
Specific objectives (SO)
Precise definition (of change)
Should be more specific than the investment priority about target groups, problem area or structure, procedures, institutions, etc. it seeks to change
Should logically link to the proposed actions
Measurement (result indicators)
Types of actions
The most appropriate interventions to achieve SO (the right "action mix"?)
The choice based on an analysis of problems
Measurement of their outputs (output indicators)

113. The ex-ante should check for a clear intervention logic
Clarifies the intervention logic
Demonstrates causal links between:
Types of actions -> outputs -> intended results / specific objective
Description of how planned actions will contribute to specific objectives is often lacking in first OP drafts reviewed.
Helps identifying for which results/outputs, not captured by common indicators, programme-specific indicators should be established
Output indicators
Result indicators

114. The ex-ante checks for relevant and clear indicators
Ex-ante evaluators should consult carefully draft guidance on ESF Monitoring and Evaluation and assess:
Relevance
Result indicators should capture a change in the situation of supported participants or entities
Clarity
Indicators should have a clear title and an unequivocal and easy to understand definition

115. Indicators with realistic targets and identified data sources
Ex-ante evaluators should also assess:
Target values – are they realistic?
Consult background papers for ESF target setting
Data collection
Will data be collected and reported on time?
Have necessary arrangements been put in place to collect and store micro-data of participants?
Are existing databases used as data sources?
(NB: reduction of admin. burden for beneficiaries)
Are procedures in place to ensure quality of data?

116. Evaluation during the programming period

117. Evaluation during the programming period (Art. 49)
MA shall ensure that evaluations are carried out to assess effectiveness, efficiency and impact and that they are subject to appropriate follow-up
Evaluation plan covering one or several OPs drawn up by MA or MS
EC may carry out evaluations at its own initiative.
Cohesion Policy
Regional Policy
117

118. Evaluation Plan (Art. 104)
Evaluation plan to be submitted to the MC no later than a year after the adoption of the programme(s)
By December MA shall submit to EC a report summarising evaluation findings and main outputs and results of programme, providing comments on the reported information
Cohesion Policy
Regional Policy
118

119. YEI Evaluation Art. 15(iv)(6) ESF Reg.
Two evaluations required: 1. completed by end completed by end 2018 Scope: assess effectiveness, efficiency and impact Be aware of scope of reporting requirement!

120. YEI Reporting Art. 15(iv)(4) ESF Reg.
The report in 2016, 2019 and 2024 "shall set out and assess the quality of employment offers received by YEI participants, including the disadvantaged, those from marginalized communities and those leaving education without qualifications. The Report shall set out and assess their progress in continuing education, finding sustainable and decent jobs, or moving into apprenticeship or quality traineeship.

121. Summary report
Art. 104(2) CPR "By 31 December 2022, managing authorities shall submit to the Commission, for each programme, a report summarising the findings of evaluations carried out during the programming period and the main outputs and results of the programme, providing comments on the reported information."

122. Ex-post evaluation (Art. 50 + 104)
The EC shall carry out the ex post evaluations in close cooperation with MS and MA
Examine effectiveness and efficiency of ESI Funds and their contribution to Europe 2020 strategy, taking account of the EU targets
By 31 December 2024 and for each ESI Fund, EC shall prepare a synthesis report outlining the main conclusions of ex-post evaluations
Cohesion Policy
Regional Policy
122

123. Contents of evaluation plan
Subject
Purpose (reasoning/use), scope, specific objectives
Key evaluation questions
Data sources and key methods
Timing (link to "reasoning")
Budget
Partnership
Audience
Dissemination strategy
Cohesion Policy
Regional Policy
123

124. Impact evaluation: Expert assistance at your disposal

125. Centre for Research on Impact Evaluation - CRIE

126. Counterfactual Impact Evaluations (CIE)
CRIE: What is it?
Centre for Research on Impact Evaluation Joint DG EMPL-DG JRC initiative
Established in June 2013
Support to MS and DG EMPL
to set up necessary arrangements for carrying out
Counterfactual Impact Evaluations (CIE)
of ESF funded interventions
In Ispra, Italy
In JRC… within the Econometrics and Applied Statistics (EAS) Unit EAS Main activities Composite indicators Uncertainty and sensitivity analysis Labour & Education economics Econometrics - CIE Staff
Researchers with a background in Economics and Statistics
Experience
Labour economics

127. WP2 CRIE support to MS
CRIE supports MS to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations (CIE) of DG EMPL funded interventions by:
WP2A Organizing training workshops on impact evaluation methods
WP2B Providing tailor-made advice on methodological and data issues that arise when designing, implementing and evaluating an initiative
WP2C Provide support to MS for arranging CIEs (preparation and implementation phase)

128. WP2A Are training workshops needed?
Specific ESF guidance on CIE now exists:
EC DG EMPL (2012),
“Design and commissioning of counterfactual
impact evaluations - A guide for ESF
Managing authorities” 
Other
books and handbooks
on CIE are now available 

129. WP2A Value added of training workshop
WP2A1. Problem-based learning methodology
WP2A2. Exemplar CIEs are used to illustrate the issues and challenges
WP2A3. Customized: Participants can propose examples of CIEs (they are working on or in the process of commissioning) for discussion at the workshop

130. Thank you for your attention!