1. A generic tool to assess impact of changing edit rules in a business survey – SNOWDON-X Pedro Luis do Nascimento Silva Robert Bucknall Ping Zong Alaa Al-Hamad

2. Business survey editing in the ONS Uses complex sets of edit rules to: –Check returned questionnaires (records) –Locate suspicious or unacceptable responses –Support data cleaning operations Edit sets are complex because they may: –Involve a large number of survey forms and variables –Contain a large number of edits –Define complex acceptance / rejection regions –Depend on a large number of tolerance parameters

3. Editing costs are high The estimated cost of editing is over 40% of the survey process budget Edits cause large numbers of record failures Edit failures are mainly dealt with by means of manual follow-up, re-contacting respondents

4. Aim of paper Describe a generic tool developed to assess the potential impact of changing the edits in any specified business survey Present example of application of the tool (SNOWDON-X) to large scale annual business survey

5. Edit revision strategies for efficiency saving Filtering or sub-setting –Comprises introducing a record filter which selects the records to be submitted to the full set of edits Gate widening –Consists of revising the tolerance parameters (gates) in individual edit rules, such that flagging of suspicious records for revision is less frequent than with previously used values Edit deletion –Consists of simply discarding some of the edits previously used to flag suspicious records

6. SNOWDON tool A SAS program developed first by Al-Hamad, Martín and Brown 2006 Developed to enable informed decision making when revising business survey edits Aims to “… help survey managers evaluate what savings can be achieved, at what cost to output quality, across many alternative permutations of editing rule parameters.” Limited to single variable survey, where only ‘gate- widening’ was considered

7. SNOWDON-X tool Extended funcionality when compared to SNOWDON Uses SAS IML language for increased performance Can handle all three edit revision strategies Can handle multivariate surveys Provides a wealth of summary indicators relating to: –Expected savings achievable by edit revision –Expected bias to survey results, both overall and per variable –Information on performance of individual edit rules / variables Simple to run, once data have been properly organised

8. Basic scenario Previous survey data available in two versions –Unedited (raw) – at point of capture or prior to any editing –Edited (clean) – at point of publication or after all editing Edit rules used to clean previous survey data are known Key idea of SNOWDON-X tool 1.Increase tolerance of some edits (or delete or introduce filter if necessary) 2.Calculate indicators of impact of changes to edits 3.Repeat 1. and 2. until expected savings achieve specified level or quality measures reveal unacceptable bias

9. Key assumptions behind approach 1.Future survey edition will behave similarly to previous survey 2.Edited data from previous survey edition are ‘clean’ or error free 3.Changes to ‘raw’ data in previous survey edition were due to error correction, i.e., any values changed between capture and final were ‘wrong’ 4.Once a record is flagged for clerical revision, all errors it contains will be located and corrected

10. What is required to run SNOWDON-X? SNOWDON-X Previous period unedited data Previous period edited data Original edits Revised edits Link between edits and variables

11. What is the output of SNOWDON-X? SNOWDON-X Descriptive indicators for the data set under analysis Indicators about individual variables and edits Indicators of impact due to changes to edit rules Modified ‘output’ survey data set

12. Core indicators

13. How to target edits for revision 1.Select most commonly used form type 2.Select edit failing largest proportion of records within each form type 3.Relax edit parameters to reduce proportion of failed records while keeping bias low 4.Repeat 2. and 3. for each form type until further savings are minimal or bias increases above specified threshold 5.Repeat for all relevant form types

14. ABI/2 (Retail questionnaire) – Number of failing records on original and revised edits

15. Results - applying SNOWDON-X to ABI/2 (Retail questionnaire)

16. Results from applying SNOWDON-X to ABI/2 Sector (questionnaire) Number of respondents Number of respondents failing validation Savings (number of questionnaires) Maximum relative bias (over all questions) Original edit rules Revised edit rules Catering1,71260654858 (9.6%)0.65% Retail3,8093,1072,934173 (5.6%)0.21% Motor Trades1,71681072090 (11.1%)0.31% Service Trades5,1081,1621,09567 (5.8%)0.04% Wholesale3,4711,8151,705110 (6.1%)0.29% Property1,06236335013 (3.6%)0.05% Production & Construction5,8262,3482,242106 (4.5%)0.53%

17. Results summary Overall expected saving for ABI/2 ≈ 6% of previously edited records Largest expected bias occurs in Catering sector (0.65%) where a saving of 58 (9.6%) records was made Highest expected saving was made in Motor Trades sector (11.1%), with an expected bias of 0.31%

18. Conclusions Generic tool developed to assist edit revision –Successfully applied to ABI2 –Currently being applied to two monthly surveys SNOWDON-X tool enables focus on edit revision, not programming for calculating quality and savings indicators Further development required for: –Impact on standard error estimates –Improved usability