1. IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement Pete Brodie ONS, UK

2. Overview Background Drivers Questionnaire requirements Delivery dilemma Solution and outcome New output requirement “Agile” methodology Outcome Results Next steps Conclusion

3. Background ONS delivers a very broad spectrum of statistics Financial pressures on all NSIs Difficult to react quickly to new requirements Scarcity of specialist skills

4. Drivers Changes due to ESA 10 and BPM6 - updated regulations for producing National Accounts Changes in the labour market – the way people are employed

5. Questionnaire requirements Early in 2014 legislative changes required substantial changes to most ONS business statistics First tranch of 28 Finance Surveys Total redesign of questionnaires Additional questions Covering new sectors of the economy

6. Delivery dilemma Standard approach for DCM uses multiple waves of cognitive interviews with responders Transcription of interviews Analysis using thematic framework Detailed report of issues and recommendations Time and available resource – impossible!

7. Solution and outcome I Brand new approach needed trialled for two surveys Condensed usual waves of testing into one week “stints” Three stages to each stint MondayTuesdayWednesdayThursdayFriday Stage 1Stage 2Stage 3 FieldworkCollaborative meetings – DCM, Clients, Stakeholders Changes agreed

8. Solution and outcome II Traffic light document

9. Solution and outcome III Extensive changes implemented for two surveys – agreed between all stakeholders Real improvements made All questions rated “green” at end Client feedback very positive Re-use methodology with some tweaks

10. New output requirements Autumn 2013 demand for measurement of the number of employees on “Zero Hours” Contracts Definition unclear No information from businesses Promise to produce estimates early in 2014 Conventional approach of qualitative and quantitative research likely to take around three years

11. “Agile” methodology I Staged approach – pilot followed by live survey Combined qualitative and quantitative work Novel (for ONS) approach of telephone survey of businesses Used flexible “Statistical Computing” for all quantitative production

12. “Agile” methodology II Pilot survey of around 1000 businesses (30 strata) Supplemented frame information with contact numbers Collaborative working between different parts of the office In field for four weeks achieved 60% response Qualitative interviews with selected responders Bespoke code for editing, outlier detection and estimation

13. “Agile” methodology III Quantitative estimates from pilot used to optimise survey design for main survey – aim for 10% cv at top level Questionnaires updated using qualitative research Main survey of 5000 businesses included all pilot sample and 4000 new members In field for six weeks achieved over 60% response

14. Outcome Published results on 30 th April – first business survey estimate of the number of live contracts of 1.4 million (c.v. 8.5%) Greater understanding of term -“contracts with no guaranteed minimum hours” Comparison with estimates from Labour Force Survey Published some more detailed estimates Some data available for longitudinal analysis

15. Results I Proportion of business using ZHC by size

16. Results II Proportion by broad industry

17. Next Steps The LFS questionnaire was updated in the light of findings and to answer detailed demographic questions Follow up business survey of 5000 now in the field Considering longitudinal design Further embedded qualitative work Considering longer term strategy

18. Conclusions These two examples illustrate what is possible when using an agile approach. Too often constrained by artificial boundaries. Opportunities for development of staff and methods through collaborative working. Necessity is the mother of invention!

19. Thank you! Any questions ?