Standard measures and variables Paul Lambert, University of Stirling Presentation to the Scottish Civil Society Data Partnership Project (S-CSDP), Webinar 3 on ‘Dealing with data: Using standard measures and variables and linking together datasets’ 10 Mar 2016

The importance of standard measures and variables “No man is an island” (Donne, 1624) In social research, we can draw upon a vast array of previous operationalisations of measures, and take advice from influential organisations/researchers on the optimal ways of constructing measures (i.e. ‘standards’) Scientific importance: Maximise replicability, consistency, reliability, validity and prospective impact of research results Pragmatic importance: Save time and energy by drawing on existing resources about important measures – see ‘CSDP workshop 1’ for notes on measures about socio-economic circumstances; voluntarism; protected characteristics S-CSDP, 10 Mar 20162

(a) Data on standards: UK NSI’s ONS guidance: mmesandservices/harmonisationprogramme mmesandservices/harmonisationprogramme Scottish government information: Methodology/Classifications Methodology/Classifications ADLS and P-ADLS: S-CSDP, 10 Mar From CODE briefing ‘How has ethnic diversity changed in Scotland?’ y/code-census-briefing-scotland_v2.pdf

(a) Data on standards: Secondary surveys...any measure you’re thinking of will probably have been used in a previous large-scale social survey [e.g. ‘Survey question bank’ at UKDS] …it’s nearly always better to re-use a measure/format that was piloted and chosen by experts, than invent a new one …secondary surveys also exhibit good practice in documentation and metadata provision S-CSDP, 10 Mar e.g. BHPS documentation online, at s/documentation

(a) Data on standards: Cross-national standards Academic literature with recommended approaches NSI’s and cross-national agencies with recommendations for comparisons, e.g. S-CSDP, 10 Mar Hoffmeyer-Zlotnik, J. H. P., & Warner, U. (2014). Harmonising Demographic and Socio-Economic Variables for Cross-National Comparative Survey Research. Berlin: Springer.

(a) Data on standards: Academic research advice on standards Methodologists tend to argue: – (e.g. Bulmer 2010; Dale 2006) – Use an existing standard unless you have a compelling reason not to – Use sensitivity analysis to operationalise, compare and document a few plausible measures – Provide crystal clear information on the standards used …but many academics ‘do their own thing’ in research and disregard standards… S-CSDP, 10 Mar 20166

(b) Using standards: ‘Data management’ issues Ideally, the construction of standard measures should be… Clearly documented (e.g. with command ‘syntax’) Consistent with published recommendations Linked to published metadata (e.g. using an ‘index file’ or ‘translation matrix’) In some situations, a standard measure isn’t plausible, but may be adapted (& should be documented) (e.g. due to sparse representations in key categories) S-CSDP, 10 Mar Operationalising many alternative socio- economic measures makes sense – but linking the data and metadata is not easy!

(b) Using standards: Data analytical issues ‘Equivalence’ considerations – ‘Measurement equivalence’ = trust the measure intrinsically – ‘Meaning’ or ‘functional’ equivalence = relative meaning, within the national/temporal/sample context (e.g. use ‘arithmetic standardisation’) Contextual considerations – Are there other important correlated factors? – Interaction terms? S-CSDP, 10 Mar Example: Highest educational qualification is a particularly difficult concept to analyse appropriately because of its strong relation to birth cohort and gender

Summary: Attention to standards is worthwhile… Time-saving, improve quality, lessen risks of errors..all for the cost of a small amount of work in checking and trying to use relevant recommendations S-CSDP, 10 Mar References cited Bulmer, M., Gibbs, J., & Hyman, L. (Eds.). (2010). Social Measurement through Social Surveys: An Applied Approach. Aldershot: Ashgate. Dale, A. (2006). Quality Issues with Survey Research. International Journal of Social Research Methodology, 9(2), Hoffmeyer-Zlotnik, J. H. P., & Warner, U. (2014). Harmonising Demographic and Socio-Economic Variables for Cross-National Comparative Survey Research. Berlin: Springer.