1. 1 Selective data editing Development & implementation Q 2010 Helsinki Jörgen Svensson Process Owner Statistics Sweden (SCB)

2. Standardization at SCB Decentralized production Development of CBM:s Editing costly, 33% of budgets Data collection departments, 2006 Standardization – the Lotta project, in 2006 22

3. 3 Nine case studies Purpose of the project: Try using selective data editing What is the potential gain using the method? Would it be possible to develop and use a common tool?

4. 4 Some results from case studies Survey Reduction % Short term employment, private sector60 Business activity indicators50 Price indices in producer & import stages50 Short term statistics, wages & salaries, private sector 40 Wage & salary structures in the private sector25 Foreign trade (5) Structural business statistics---

5. SUSPICION SUSP(j, k) = Suspicion of variable j for unit k SUSP(j, k) = 0 if variable value falls within acceptance interval SUSP(j, k) → 1 as value deviates from acceptance limit 0 ≤ SUSP(j,k) ≤ 1

6. POTENTIAL IMPACT POTIMP = Potential impact POTIMP is weighted absolute difference between observed and predicted value : POTIMP(j,k,d) = for variable j, unit k in domain d w k is sampling weight,  k (d) is domain indicator SELEKT supports several ways to establish predicted value: from time series data and from cross sectional analysis within homogenous groups of units

7. Flagging suspected errors log(Potential impact) log(Suspicion) 20 Flagged

8. LOCAL SCORE Local (item) score LScore (j,k,d): LScore (j,k,d) = SUSP(j,k)*|POTIMP(j,k,d)|*Cello(j,d) Cello(j,d) is inversely proportional to the standard error based on previous data

9. GLOBAL SCORE Global (unit) score GScore(k) is obtained by aggregation of local scores LScore (k, j, d) → LScore (k, j) → GScore(k) → = Summation, Euclidian Summation or Maximum Only those units with GScore larger than a pre-decided threshold are followed up

10. SELEKT, EDIT and process data 10

11. Implementation of SELEKT So far three surveys: Business activity indicators Wage & salary structures in the private sector Commodity flow survey 11

12. 12 Documentation A General Methodology for Selective Data Editing jorgen.svensson@scb.se anders.norberg@scb.se