Update on the MIS risk assessment notes

Slides:



Advertisements
Similar presentations
ESF - Monitoring: Q&A Andy Fuller and Nirina Rabemiafara
Advertisements

1 Owner-Occupied Housing Summary of the pilot Item 5 of the Agenda D4 – Price Statistics HICP Working Group Luxembourg October 2007.
1 Ex-ante evaluations of ESF operational programmes Budapest 26 th September 2013 Kamil Valica Unit A.3 Impact Assessment and Evaluation DG Employment,
ESS Slide 1 Quality assessment of MEHM in SILC Eurostat Unit F5 “Health and Food Safety Statistics” 4 th meeting of the Task Force on Health Expectancies.
Module V Creating awareness on validation of the acquired competences
Eurojust cases involving crimes against children
Weighting issues in EU-LFS
ESF transnational calls – Member State plans
Support in the implementation of the REFIT actions for ELD phase 2
ESF Data Support Centre Data collection and validation support for the management of the ESF Andy Fuller ESF Support Centre Alphametrics Ltd.
Update on the UOE 2012 data collection
Ag.no. 15 Lessons from the 2015 A65 exercise
ESF - Monitoring: Q&A Andy Fuller and Nirina Rabemiafara
2.1. ESS Agreement on Learning Mobility (IVET & Youth)
CEF eID SMO The use of eID in eHealth
Ex-ante evaluation: major points and state of play
ESS Security and Secure exchange of information Expert Group (E4SEG) DIME/ITDG Item 8 ESS Security Assurance Pascal Jacques ESTAT B2 Local Security Officer.
Update of Practical guidance (Annex D) & FAQ Guide
ESF Support Centre Alphametrics Ltd. & Applica Sprl.
Marine Strategy Framework Directive: Transposition and Implementation
State of play Article 5 reports
ESF FINANCIAL EXECUTION ESF Technical Working Group Meeting June 2018
Education and Training Statistics Working Group
Agenda Item 2.1 SES 2014: follow-up
1.
Survey on representative sampling
Representative sampling questionnaire
Marine Strategy Framework Directive: Transposition and Implementation
Marine Strategy Framework Directive: Status of reporting
LAMAS Working Group 29 June-1 July 2016
ESF FINANCIAL EXECUTION ESF Technical Working Group Meeting June 2018
State of progress with transition to new Standard Data Form
2a. Status of WFD reporting
Reporting – Art 17 of HD and Art 12 of BD
Item 7.1 Implementation of the 2016 Adult Education Survey
Item 8.1 Implementation of the 2016 Adult Education Survey
2b. Status of WFD reporting
Programme adoptions Cohesion Policy:
Ag.no. 15 Lessons from the 2016 A65 exercise
Representative sampling Overview of the questions received by the ESF Data Support Centre Alphametrics Ltd. & Applica Sprl. Brussels, 13 March 2015.
Item 7.1 – Overview of 2012 UOE data collection
ETS Working Group meeting 24-25/9/2007 Agenda point 7 CVTS3 brief update /09/ 2007 ETS working group.
State of play: data transmission, validation and dissemination
Debrief of Learning Seminar on Youth Employment Initiative
Update on legal issues Strategic Coordination Group
Update on the status of RBMP reporting
Item 4.1 – Overview of 2014 UOE data collection
YEI reporting: state of play
ESF FINANCIAL EXECUTION ESF Technical Working Group Meeting June 2018
Item 4.3 – Repeal of CVTS legal acts
Findings of the MIS survey and work programme for 2015
3.6. Impact of population and housing census results on population stocks and on LFS and SILC–follow-up DSS Meeting September 2012.
Labour market statistics- State of play
Update on legal issues Strategic Co-ordination Group 7-8 May 2009
Data collection and validation support for the management of the ESF
Legal and implementation issues update
Meeting of Water Directors State of transposition and implementation
State of Play RBMPs and WISE reporting (9/07/10)
FISIM State of play Agenda Item 3.
Update on legal issues Strategic Coordination Group 23 February 2010
Update on legal issues Strategic Coordination Group
Update on status of reporting and validation process
Doc.A6465/16/03 Ag.no.16 A65 country manuals
Doc.A6465/14/04 Ag.16 A65 country manuals
IT security assurance – 2018 and beyond Item 2 of the agenda DIME/ITDG Steering Group June 2018 Pascal JACQUES ESTAT B2/LISO.
Transposition and implementation of WFD
LAMAS Working Group June 2015
Review of BWD reporting for 2009 season (Item 2)
Modernisation of Validation in the ESS Collaboration with countries
EDAMIS3: CURRENT STATUS
Presentation transcript:

Update on the MIS risk assessment notes ESF Support Centre Alphametrics Ltd. & Applica Sprl. Brussels, 10 June 2015

Risk assessment notes Based on the MIS fiches received by end-February 2015, risk assessment notes were produced to identify any potential risks to data quality that arise from the MIS developed by each MA. March 2015: the risk assessment notes were sent by the EC country desks to the MAs. Up to now, feedback was received from 9 out of 41 MAs.

Main issues (re-cap) MIS structure and development Not ready for YEI reporting Need of testing MIS (recommended to provide support) Definitions of indicators Inconsistent application Not in line with EC guidance (“migrants, foreign background, minorities”, “other disadvantaged”, and use programme-specific indicator instead) Data collection Arrangements with data protection authorities not in place Participants refusing to provide information will not be counted Use of administrative registers: delays, entry/exit dates […] 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Main issues (re-cap) Data validation Data aggregation […] Only at one level (e.g. only at MA level, only at beneficiary level) No automated validation checks Data aggregation No automatic aggregation by IP and region Lack of (automated) measures to ensure reporting includes retrospective changes Delays in alternative aggregation or not possible at all 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Main issues (re-cap) Data storage Representative sample […] Full micro-data set not available Representative sample Sample include incomplete records, non-supported participants Samples drawn at lower level (e.g. beneficiary) 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed MIS development Readiness of MIS for YEI reporting Regional MIS already operational (even if not finalised yet) Until MIS ready, data collected/stored in PES system YEI implementation has not started yet (system ready by June) Development and testing phase Working groups set up for stages of development of IT system New MIS is being tested IT user manual + trainings 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Definitions – not in line with ESF guidance Other disadvantaged: homeless not included Will not be included to avoid duplication – refer to updated guidance Migrants, foreign background, minorities: groups not covered recommended definition not applied Amended definition Definitions – inconsistent application Other disadvantaged: open definition The definition, although with guidance provided will remain open (i.e. risk only partially addressed) 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Data collection Discussions still ongoing with data protection authorities Permission should be granted by the time MIS is ready. Data protection authority has checked the data collection forms, the guidance note and the legislation. Refusal to provide non-sensitive data leading to non-eligibility Individuals can be supported (if they fulfil the eligibility criteria). Recommendation: MIS should be able to record incomplete records (grand-total of supported participants). Recommendation will be followed. […] 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Data collection […] Administrative data not reflecting situation on entry/exit, or obtained with a delay Pilot data collection: project start date = entry date for all participants but adjustments might be made following evaluation of the pilot Registers not updated on a daily basis but provide high quality data Collection of sensitive data based on anonymous forms with no participant ID Personal codes generated by the MIS might be included in the anonymous form (pending data protection agency authorisation) 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Risks addressed Data aggregation Alternative aggregations In-built filters cover a wide range of variables Data for ad hoc requests provided by Statistical Office Obtained with some delay as some data are stored in the NSI Automated mechanism to identify data that have been retrospectively modified Dates recorded in the system and identification of updated data 34 YEI OPs in 20 MS: BE(2), BG(1), CZ(1), IE(1), EL(1), ES(1), HR(1), HU(1), CY(1), IT(1), LT(1), LV(1), FR(13), RO(1), PT(1), PL(1), SE(1), SI(1), SK(1) and UK(2). MIS not fully operational for the YEI report  potential risk of errors in the treatment/reporting of participation records (e.g. if validation/ aggregation procedures are not in place yet). It is necessary to test the new monitoring system (e.g. by using pilot projects) before launching it. Moreover, it would be useful to provide support (e.g. IT user manual, FAQ, IT support desk etc.) to all organisations using the system for the first time.   

Feedback is precious! In addition to the risks identified in the RA notes, many issues are still pending or waiting for further clarification…  Please send your replies/comments/updates: 1st round consultation: those who have not reacted to the RA note yet 2nd round consultation: those who already reacted to the RA note and for whom an updated RA note was sent No feedback received on important risks identified, particularly automated data validation

ESF data support centre Advice and guidance on: Methodological support on monitoring information systems Clarification regarding data collection, recording, validation and storage Guidance regarding transmission of structured data to the EC using SFC2014 (technical queries > SFC support team) 9.00 a.m. – 5.00 p.m. Monday-Friday E-mail: support@esfsupport.eu Phone: +44 1480 702003