AN EXAMPLE OF COOPERATION & SOME WIDER ISSUES Ian Plewis (Bedford Group, Institute of Education) & Stephen Morris (Social Research Division, Department for Work & Pensions)

OVERVIEW Government research identifies important methodological issues These issues often/sometimes ignored or assumed not to be a problem –exception ONS work on methods & quality Where methods research takes place no government forum for exchange exists Child outcomes work (IoE/DWP) – a methods project - to illustrate the problem We suggest why methods often not addressed & thoughts on potential solution

THE RESEARCH A feasibility study Are adult outcomes for individuals (parents) linked to (or caused) by changes in childhood circumstances? Effects of change in household income (household deprivation & parental employment) in childhood on individual’s adult outcomes Pathways through which changes take effect?

POLICY CONTEXT Government child poverty targets – end child poverty by This research informs development of policies to meet targets Marginal £ - what be the most effective way of addressing child poverty –Income transfers? –Services? Our focus on changes in income thus whether transfers are likely to be effective

WHY NEW RESEARCH? Not much UK research looking at change (Our interest is in establishing causality) Tends to focus on short-range educational outcomes Focus on single data sources Most useful sources – birth cohorts – have limitations Can data sources be combined to better address questions of relevance –What assumptions do we have to make? –What statistical models could be used?

THE PROBLEM Most quantitative social research has to deal with methodological issues. Some of these issues are ‘cutting edge’ and need to be addressed within a research council project. Some are ‘standard’. But many fall between these two extremes and tend to get ignored in Government funded projects

MEASUREMENT ERROR Suppose we want to estimate a regression model with a measure of income as one of possibly several explanatory variables. We know that measures of income are unreliable and we also know that this unreliability, if ignored, can lead to biased estimates of coefficients of interest. We should correct for measurement error because not to do so could lead to misleading inferences for policy. But such corrections are not standard and the problem is often swept under the carpet..

NON-RESPONSE Unit non-response and attrition are features of all longitudinal datasets. Certain kinds of cases tend to be lost – at the outset and over time. Ideally, we should carry out sensitivity analyses of some kind to establish whether losses from the sample are likely materially to affect policy conclusions. But these analyses take time and resources

INCOME MISSING Data are often missing for income even when other variables are measured. And some surveys do not collect income data. Imputation is a way round some of these difficulties. But there are a number of imputation methods – how does their application affect our conclusions?

IMPLICATIONS FOR POLICY Measurement error/non-response = biased estimates Biased estimates can lead to bad policy decisions Example (hypothetical): –Under-estimate of effect of change in income on child outcomes –Assume income transfers less effective –Direct policy toward services –Sub-optimal policy response

WHY MIGHT THIS HAPPEN? No cross-government forum for exchange of information on methods research Within spending departments not always easy to make the case for methods research –Benefits of methods research diffuse –Not priority for policy customers Researchers don’t always appreciate implications Time – complex problems take time to solve

WHAT COULD BE DONE? Establish a GCSRO/NCRM methods group (representatives: depts., ONS, ESRC & NCRM) Forum for raising methods related issues with policy focus – disseminate existing work from ONS, depts. & academics/researchers Identify gaps and commission – requires funding –Policy relevance –Wider application Dissemination – publications, seminars and training courses CASE studentships

CONTACT DETAILS Ian Plewis – Institute for Education (T) (E) Stephen Morris – Department for Work & Pensions (T) (E)