Survey on representative sampling ESF Support Centre Alphametrics Ltd. & Applica Sprl. Brussels, 13 November 2015

Background Scope Objectives 2 common output indicators ESF longer-term result indicators YEI longer-term result indicators Objectives To raise awareness on the requirements To assess the state of preparedness To identify potential issues

The survey Launching: beg-July, ongoing ( end-Nov) Combination of open/multiple choice questions 54 MAs contacted, 24 replies received, 21 respondents 9 final 12 clarification pending 3 will not participate

Use and coverage of surveys Use of surveys to collect data In the majority of cases (16), data based on surveys for some indicators (especially CRI7 on improved labour market situation)  Other indicators: either based on registers or data collected on entry (homeless/from rural areas) In 3 cases surveys planned for all indicators In 1 case, no surveys planned (all data from administrative registers) 1 MA still to be decided Coverage of surveys Surveys based only on representative samples, in half of the cases (10) Surveys based on all participants, in a few cases (4) Surveys based on both, in a few cases (5)

Data collection approach for each indicator   Survey - full population Survey - samples Survey (mix) Registers On entry To be decided No reply - N/a Total respondent MAs Common output indicators COI18 2 1 3 9 5 21 COI19 7 Common longer-term result indicators CRI6 4 11 - CRI7 6 12 CRI8 CRI9 YEI common longer-term result indicators YEI RI10 8 YEI RI11 10 YEI RI12

Survey and sampling design Authority responsible for drawing the samples Mainly independent expert institution Also IB and NSI Sampling method: very little information at this stage Margin of error In 2 cases <=2 pp, in 2 cases 2-5 pp Still to be decided in many cases Coverage errors: very little information at this stage (…)

Survey and sampling design (…) Non-sampling errors Methods used to minimise Ensuring clear formulation of questions Questionnaires adapted to target group Preparation of interviewers Testing of questionnaires On spot verifications Definitions and translations of indicators available for participants In many cases, determined by the external contractor/institution drawing the samples

Data collection Authority responsible for data collection External body in many cases Also NSI, MA, IB, and beneficiaries Method for data collection Phone (1), email (1), online forms (3) Combination of methods (4) Still to be decided in many cases (…)

Data collection (…) Data collection schedule Minimum requirements   Minimum requirements Annual More frequent On entry Other To be decided No reply N/A Total respondent MAs COI18 1 2 8 6 3 21 COI19 4 CRI 5 - YEI RI 9

Data storage Storage of LTRI data In the original database with all the micro-data in many cases (9) In some cases in a separate database (3) MA only store aggregated data (4) Depending on the data (2) Still to be decided (3) Ensuring the correct reference population (including exit dates of participants) and data breakdown at the level of IP Requirements for external organisations laid down in the specifications Full set of micro-data transferred to organisation doing the sampling Linking databases by using personal IDs

Main difficulties Low response-rate, particularly for hard to reach participants Cost of surveys Precision requirements published in June 2015 not applied in surveys for YEI which had already been launched Timeliness of surveys, possible outdated contact details Problems to identify and contact homeless participants Rural areas: inaccurate or incomplete postcodes, no correspondence between DEGURBA and national register for postcodes

Budget Very little information about the planned budget (only 4 replies) Evaluation Surveys will be used to collect information for (counterfactual) evaluation purposes by 7 MAs. Not used by 3 MAs.

Conclusions No comprehensive picture Few responses Outsourcing of tasks and arrangements still ongoing in many cases No misinterpretation of requirements or deviation from guidance was identified (3 cases pending clarification) A mix of approaches, no predominant patterns (except CRI07)  Participation is encouraged!

ESF data support centre Advice/guidance available only until December 2015! Methodological support on monitoring information systems Clarification regarding data collection, recording, validation and storage Guidance regarding transmission of structured data to the EC using SFC2014 (technical queries > SFC support team) 9.00 a.m. – 5.00 p.m. Monday-Friday E-mail: support@esfsupport.eu Phone: +44 1480 702003