Update on the MIS risk assessment notes ESF Support Centre Alphametrics Ltd. & Applica Sprl. Brussels, 10 June 2015

Risk assessment notes Based on the MIS fiches received by end-February 2015, risk assessment notes were produced to identify any potential risks to data quality that arise from the MIS developed by each MA. March 2015: the risk assessment notes were sent by the EC country desks to the MAs. Up to now, feedback was received from 9 out of 41 MAs.

Main issues (re-cap) MIS structure and development Not ready for YEI reporting Need of testing MIS (recommended to provide support) Definitions of indicators Inconsistent application Not in line with EC guidance (“migrants, foreign background, minorities”, “other disadvantaged”, and use programme-specific indicator instead) Data collection Arrangements with data protection authorities not in place Participants refusing to provide information will not be counted Use of administrative registers: delays, entry/exit dates […] 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Main issues (re-cap) Data validation Data aggregation […] Only at one level (e.g. only at MA level, only at beneficiary level) No automated validation checks Data aggregation No automatic aggregation by IP and region Lack of (automated) measures to ensure reporting includes retrospective changes Delays in alternative aggregation or not possible at all 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Main issues (re-cap) Data storage Representative sample […] Full micro-data set not available Representative sample Sample include incomplete records, non-supported participants Samples drawn at lower level (e.g. beneficiary) 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed MIS development Readiness of MIS for YEI reporting Regional MIS already operational (even if not finalised yet) Until MIS ready, data collected/stored in PES system YEI implementation has not started yet (system ready by June) Development and testing phase Working groups set up for stages of development of IT system New MIS is being tested IT user manual + trainings 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Definitions – not in line with ESF guidance Other disadvantaged: homeless not included Will not be included to avoid duplication – refer to updated guidance Migrants, foreign background, minorities: groups not covered recommended definition not applied Amended definition Definitions – inconsistent application Other disadvantaged: open definition The definition, although with guidance provided will remain open (i.e. risk only partially addressed) 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Data collection Discussions still ongoing with data protection authorities Permission should be granted by the time MIS is ready. Data protection authority has checked the data collection forms, the guidance note and the legislation. Refusal to provide non-sensitive data leading to non-eligibility Individuals can be supported (if they fulfil the eligibility criteria). Recommendation: MIS should be able to record incomplete records (grand-total of supported participants). Recommendation will be followed. […] 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Data collection […] Administrative data not reflecting situation on entry/exit, or obtained with a delay Pilot data collection: project start date = entry date for all participants but adjustments might be made following evaluation of the pilot Registers not updated on a daily basis but provide high quality data Collection of sensitive data based on anonymous forms with no participant ID Personal codes generated by the MIS might be included in the anonymous form (pending data protection agency authorisation) 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Data aggregation Alternative aggregations In-built filters cover a wide range of variables Data for ad hoc requests provided by Statistical Office Obtained with some delay as some data are stored in the NSI Automated mechanism to identify data that have been retrospectively modified Dates recorded in the system and identification of updated data 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Feedback is precious! In addition to the risks identified in the RA notes, many issues are still pending or waiting for further clarification…  Please send your replies/comments/updates: 1st round consultation: those who have not reacted to the RA note yet 2nd round consultation: those who already reacted to the RA note and for whom an updated RA note was sent No feedback received on important risks identified, particularly automated data validation

ESF data support centre Advice and guidance on: Methodological support on monitoring information systems Clarification regarding data collection, recording, validation and storage Guidance regarding transmission of structured data to the EC using SFC2014 (technical queries > SFC support team) 9.00 a.m. – 5.00 p.m. Monday-Friday E-mail: support@esfsupport.eu Phone: +44 1480 702003