The Application of Statistical Matching to the 2010 ESF Leavers Survey Rhys Davies Wales Institute for Social and Economic Research, Data and Methods, Cardiff University Gerry Makepeace Cardiff Business School, Cardiff University Findings are not in the public domain and are not for further dissemination

Overview of 2010 ESF Survey Aim – understand the characteristics and outcomes of those participating in ESF funded projects Methodology - telephone survey of people who were identified as having left an ESF project during 2010 Survey conducted Jun/July 2011 Interviews achieved with 7,509 participants (50% response rate) P2/P3 Convergence , P1/P2 Competitiveness Programme Similar survey conducted in 2010 among 2009 Leavers Previous and current activity, why did ESF, withdrawal from ESF, skills gained from ESF, educational attainment (pre/ESF/post), current employment, perceived additionality

Transitions in Main Activity: Results from the 2009 Survey Main activity before attending ESF course Current main activity at time of survey Paid employment Education and training Unemployed Economically inactive Total 8.3 1.5 1.1 0.6 11.4 5.6 3.9 3.0 1.3 13.8 21.8 7.4 21.0 6.1 56.4 2.8 2.6 10.7 17.5 38.7 14.3 28.0 19.9 100 Positive Transition No Transition Negative Transition

The Labour Force Survey as a Control Group ESF Survey provides no control group how do these transitions in to employment compare to those observed among wider population? what would ESF participants have done in the absence of ESF Use the UK Labour Force Survey to provide a control group against which the effect of participating in ESF can be evaluated Respondents to the LFS are asked what they were doing one year earlier so can look at transitions in economic activity over a period of 12 months Questions in ESF Survey and the coding of data are designed to align with definitions used in the LFS

Propensity Score Matching Use Propensity Score Matching to extract people from the Labour Force Survey who share similar characteristics to ESF participants Matching variables include age gender partnership status (couple/single/living at parental home), family status (dependent children under age of 18), health (work limiting health condition) educational attainment (NQF equivalents) ethnicity local employment (rate of employment among non-student population – allocated to decile groups)

Some Practical Issues The LFS data pre-dates the information from the ESF survey Introducing local labour market indicators requires access to detailed geographical identifiers Time elapsed between pre-ESF and current activity varies among ESF respondents – not set at 12 months a) the varying duration of ESF interventions and b) the different end dates OF these interventions Use career history data to identify activity 1 year following pre-ESF Only variables unaffected by participation within an ESF project should be used in matching – we use some variables that are measured at the time of the surveys Unemployed more homogenous than economically inactive

Career Profiles of Previously Unemployed – 2009 Survey

12 Month Transition Rates in to Employment Among the Previously Unemployed Local Area Employment Rates (non-student population of working age) LFS Transition Rates ESF Transition Rate (2009 Survey) 1st Decile (<68.8%) 29% 40% 2nd Decile (68.8-70.7%) 32% 3rd Decile (70.7-73.3%) 33% 4th Decile (73.3-74.4%) 34% 5th Decile (74.4-75.7%) 35% 6th Decile (75.7-76.6%) 42% 7th Decile (76.6-78.0%) 41% 8th Decile (78.0-79.5%) 44% 9th Decile (79.5-80.8%) 47% 10th Decile (>80.8%) 52% Total 38%

Propensity Score Matching Results Different matching techniques to consider sensitivity of results: Nearest Neighbour Select the individual who has ‘closest’ propensity score Radius Matching Average characteristics of controls within a specified radius With and without replacement allow controls to be matched to multiple ESF respondents Adjust size of callipers the maximum acceptable difference in propensity scores Include/exclude: ESF early withdrawers - Proxy respondents from the LFS Continuous career histories from the LFS

Conclusions, Limitations and Future Directions Additional insights can be provided by PSM if ESF surveys are designed to align with other available sources of data Limitations remain can only match individuals on the basis of measurable characteristics our control group may have participated in some other form of assistance some matching variables could have varied over the period during which employment transitions were being measured Future challenges refinement of statistical matching to use longitudinal data sources understanding of what works requires project level analysis