ESRC UK Longitudinal Studies Centre A Framework for Quality Profiles Nick Buck and Peter Lynn Institute for Social and Economic Research University of Essex

ESRC UK Longitudinal Studies Centre Overview Data quality is a key input into the quality and robustness of research – and is one which researchers do not necessarily have the tools to consider systematically Work on quality frameworks aims to define the dimensions of quality – and move beyond the most obvious elements Quality profiles specify how these dimensions of quality might be measured and documented Implementation for longitudinal studies, e.g. BHPS Quality Profile

ESRC UK Longitudinal Studies Centre Quality dimensions in general Statistical institutes have sought to identify key dimensions of quality e.g. Statistics Canada Quality Guidelines identify: –relevance –accuracy –timeliness –accessibility –interpretability –coherence

ESRC UK Longitudinal Studies Centre Quality dimensions Other versions have formulated this marginally different ways, but essentially the same. Some also include cost – others argue this should be considered separately from other dimensions NLSC work has adopted StatsCan version Longitudinal studies have distinctive features, but all can still be considered within the context of the above framework

ESRC UK Longitudinal Studies Centre Relevance The extent to which the data meet the needs of users - interpreted in the broadest sense Were the right questions asked? Were they asked of the right people/ units? Were they asked in the right way?

ESRC UK Longitudinal Studies Centre Accuracy The difference between survey estimates and population parameters - total survey error Coverage error Sampling error Non-response error Instrument errors Respondent errors Interviewer errors Processing errors Overall accuracy

ESRC UK Longitudinal Studies Centre Timeliness The timeliness of survey outputs (data, documentation, reports) refers to the delay between the time reference point (or end of reference period) and the date when the outputs become available. Distinguish absolute measure (the length of the delay) from delay relative to that anticipated by users. Also interacts with relevance – are data collected at the right point for research purposes, is the right time reference period used

ESRC UK Longitudinal Studies Centre Accessibility The ease with which relevant data or outputs can be obtained, including –the ease with which the existence of data/outputs can be established, –the suitability of the form or medium through which they are accessed –any barriers to use of the outputs, such as costs or restrictions. To be assessed qualitatively and/or through user consultation exercises.

ESRC UK Longitudinal Studies Centre Interpretability The availability of supplementary information necessary to interpret and utilise outputs appropriately, including both metadata and microdata. Metadata would include details of underlying concepts, variables and classifications, definitions of derived variables, contextual data, methodology of data collection, indications of statistical accuracy and copies of all survey documents (in an appropriate form). Microdata would include indicators of sampling stratum, PSU, weights (and their components), imputation flags, field effort indicators, and so on.

ESRC UK Longitudinal Studies Centre Coherence The extent to which data/reports from different sources can be brought together within an interpretative framework and over time. For longitudinal studies, this includes both coherence across waves/sweeps of a study and coherence between the study and other studies (both longitudinal and cross- sectional). Coherence is promoted by the use of standard concepts, classifications and target populations, and common methodology amongst surveys likely to be compared and across waves of a longitudinal study. Coherence is sometimes in conflict with relevance

ESRC UK Longitudinal Studies Centre Specific relevance to longitudinal studies Sample attrition - non-response bias is important for any survey, but cumulative attrition is specific to LS. Definitions of the study population, which changes over time and between analyses – e.g. deaths, births, emigrants, immigrants and so on. Item non-response – impact greater because available sample may depend on valid response on every occasion. Changing relevance of data items and variables, as research agendas change. Long-term LS encounter conflicts between relevance and consistency. Changes in data collection and processing technology and in the research team for long studies have implications for compatibility and consistency.

ESRC UK Longitudinal Studies Centre The Role of Standard Quality profiles Documenting in an accessible manner the main aspects of data quality for each study (and, as a result, enabling users to make more appropriate uses of the data); Identifying priorities for methodological research, to fill gaps where critical aspects of data quality cannot be properly documented; Identifying priorities for remedial work on a study, where the quality profile suggests deficiencies; Identifying priorities for future developments of a study, where the quality profile suggests particular strengths and opportunities; Raising awareness of data quality issues, and hence contributing to quality improvements.

ESRC UK Longitudinal Studies Centre Quality profile template Since one aim of the quality profile is to document in an accessible manner, there is strong case for adopting a standard format – hence NLSC adopted quality template below, to be used for all ESRC supported studies. There may be issues of implementation for individual studies, but in general all headings should be relevant, and if they not it will be important to say so. Also, if no information available, important to say so. Measures based on output quality and process quality

ESRC UK Longitudinal Studies Centre Contents of standard template 1.Statement of Core Research Purposes – to clarify relevance issues 2.Overview of the Survey Design 3.Sample Design – both initial wave and following rules 4.Content of Data Collection Instruments – focussing on underlying principles and procedures for developing questionnaires 5.Data Collection – both a documentation of process, e.g. interviewer selection and allocation, quality control procedures, and of outcomes – unit and item response

ESRC UK Longitudinal Studies Centre Contents of standard template -2 6.Data Preparation – evaluation of coding and editing and other related procedures 7.Statistical Adjustment Procedures – documenting and evaluating weighting and imputation methods 8.Documentation and Data Accessibility 9.Data Usability and Interpretability – e.g. how consistently organised data is, how data includes information required to assess quality – e.g. non- response indicators, design elements 10.Coverage Error – assessment of the extent of under or over-coverage, both cross-sectional and longitudinal

ESRC UK Longitudinal Studies Centre Contents of standard template Sampling Error – including both sampling variance, and bias arising from e.g. longitudinal sampling rules 12.Non-Response Error – biases arising from initial non-response and attrition 13.Measurement Error – evidence available on error in measures of gross flows and e.g. retrospective recall 14.External Comparisons – evidence available on comparisons between this study and others

ESRC UK Longitudinal Studies Centre Other issues for quality profiles Document whole study or individual waves or sweeps? Keeping them updated Relevant issues which do not fit directly into the framework – e.g. ethical issues and confidentiality

ESRC UK Longitudinal Studies Centre Implementation BHPS profile completed for waves 1 to 10, available at - this is (probably) the first full systematic quality profile for a UK survey. Quality profiles are in preparation for birth cohort studies Further documentation on National Strategy Committee pages: ence/quality-profiles.php