 Ewen McCaig Scottish Household Survey Review 2005 Ewen McCaig McCaig Services

 Ewen McCaig External Review To assess whether the SHS is fit for purpose now, and to ascertain if, and in what form, it should continue  Survey Purpose  Topic Coverage  Data Issues  Survey Design  Output and Dissemination  Project Management and Control  Future Developments of the Survey

 Ewen McCaig Scope  Not an evaluation of performance or efficiency  Not an ‘independent’ review - purpose was to support a broader SE review process  Options required, rather than recommendations  Strategic, not detailed  Topic cover to continue to focus on transport, communities and local government  Recognise and support external interests but give priority to SE requirements

 Ewen McCaig Method  Report to Review Team  Document review  Consultation  Topics and dissemination with users (SE and external)  Design and management with SE stakeholders and users

 Ewen McCaig Consultation  Intended to give all a chance to contribute:  SE staff,  other public sector,  academics,  voluntary sector  consultants  , phone and important SE contacts were face- to-face

 Ewen McCaig Consultation  Contact list compiled from  Scottish Executive personnel identified by the SHS team  Distribution lists of SHS data and reports  Local authority contact lists  Attendees at the last SHS User Group event  Persons who had requested data or special analysis from the SHS team  The ScotStat mailing list  Persons identified during the consultation by snowballing

 Ewen McCaig Response  Consultation, not a survey, so response rate is not significant  Total s sent: 454  or postal return: 107  Telephone interview: 34  Face to face interview: 35  No contact: 33  Do not use SHS: 15

 Ewen McCaig Questionnaire Content  All existing topics had users and additional demands identified  Perception that transport was main focus not justified (13.8 out of c50 minutes)  Local authority topic now less prominent but still relevant  Communities topics would benefit from further consultation and extension  Image of survey was not always clear to users  Possible review of income and travel diary questions  Some feeling that ongoing consultation on topic cover could be improved

 Ewen McCaig Modularisation  Current interview too long and more material wanted  More material cannot be included if everything covered in every interview  Cannot be expected that identical sample sizes needed for all questions in multi-themed survey  Modularisation means covering selected questions with a part sample - eg if 50% of question time asked of 50% sample, then topic coverage increases by 25%

 Ewen McCaig Modularisation (cont)  Only way to increase cover - more material moves towards critical mass of cover, strengthens thematic identity, introduces flexibility  More elaboration and some cost implications but better value for money  Some questions would be asked of part-samples but still large numbers by Scottish standards  Main adverse impact on disaggregation of some of the findings

 Ewen McCaig Modularisation Design  Socio-demographic and topic ‘core’ content in every interview  Length of core determines residual time for topics. Illustrative example:  Average interview length45 minutes  Length if all questions asked 80 minutes  Core length25 minutes  Non-core55 minutes  Non-core time available20 minutes  Implied mean sample size for non-core questions is 5,700 per year or 11,400 bi-annually  Even 12.5% cover gives an annual sample of 1,875  Detailed design must follow consultation

 Ewen McCaig Sampling  Total sample size not covered in review - increases possible but must be practicable  Stratification with minimum sample by LA still seems sensible - allows national reporting and tends to weight sample into areas of low population  Key point is clustering - sample currently clustered in the 23 least urban LAs

 Ewen McCaig Sampling (cont)  Benefits of unclustering clear: the only potential issue is fieldwork capacity and cost  Some allowance already for cost with sample boost intended to compensate for cluster design effects  Increased cost can be offset against increased sample size required to compensate for design effects

 Ewen McCaig Benefits of Unclustering  No design effects due to clustering  No geographic gaps or bias in the data  Relative simplicity and transparency  Easy expansion to population totals for different areas  Data available for more geographic areas without bias and complex weighting options  Fewer obstacles and risks for secondary analysts  Easy utility in systems involving geographic selection and analysis  Flexibility in merging samples with other surveys

 Ewen McCaig Other Issues  Dissemination could emphasise thematic content more  Consultation on topic cover important  More emphasis on web dissemination could be considered  Find role in development of cross-sectional survey standards in Scotland