Applying Process Indicators to Monitor the Editing Process

Purpose of Process Indicators Monitor and improve the editing process Find measurement errors and their causes and prevent them in future survey rounds Communicate quality to the users

International Trade in Services Statistics on international trade in services, wages and transfers are based on a quarterly sample survey involving about enterprises and organizations. Report values of income and expenditure for about 100 service types.

Editing in the survey Selekt is used Fatal errors are always followed-up Suspicious errors are used to create a local score for each service type based on potential impact and suspicion Local scores are aggregated to a global score for each enterprise A threshold determines whether the enterprise needs to be followed-up or not

Descriptive information 2013Q12013Q22013Q3 Total number of observations Number of observations failing at least one edit rule Number of observations failing at least one edit rule and adjusted Expenditure Income 2013Q12013Q22013Q3 2013Q12013Q22013Q3 Weighted raw value sum Weighted raw value sum, failing at least one edit rule Weighted edited value sum

Overall failure rate, 2013Q Q3 I1I1 Overall failure rate. Rate of observations failing at least one edit rule.

Overall hit rate, 2013Q1-2013Q3 I2I2 Overall hit rate. Rate of observations failing at least one edit rule and adjusted.

Weighted reject ratio, 2013Q Q3 I3I3 Weighted reject ratio.

Failure rate and hit rate for each edit, 2013Q1-2013Q3 I4I5I4I5 Failure rate for each edit FR. Hit rate for each edit HR Edit rule2013Q1 2013Q2 2013Q3 FRHRFRHRFRHR Q035.05%54%0.77%60%1.93%91% Q070.12%20%0.08%67%0.08%67% Q086.14%67%6.22%65%5.66%53% Q095.32%68%5.08%64%4.50%50% Q100.40%63%0.34%23%0.39%60% Q110.89%53%0.88%24%1.05%29% Q120.74%77%0.80%77%0.64%76% Q142.13%72%2.66%67%2.24%64% Q150.20%100%0.03%100%0.10%100% Q160.64%85%1.86%89%0.80%87% Q190.47%95%0.80%84%0.72%89% Q200.17%57%0.18%29%0.10%50%

Editing rate for each variable, 2013Q1-2013Q3 I6I6 Editing rate for each variable. Expenditure Income 2013Q12013Q22013Q32013Q12013Q22013Q3 410 Computer services0.8%0.4% 463 Other services between affiliated enterprises0.4% 0.3% 442 Architectural, engineering and other technical services0.7%0.5%0.4% 143 Goods freight by road0.3%

Problems Tedious to calculate Difficult to interpret Often small amounts of data Variable specific indicators depend on random fluctuations

Data availability Dependent on available data Some thoughts on which indicators we want should be considered when setting up the system Structure of data influences difficulty to generate indicators A system that generates standard reports could be beneficial

Conclusion Standardized indicators could be beneficial Needs guidance on how to interpret and use them Will take time to establish indicators in the organization and within the management teams - What possibilities do we have to spread them to the organization?

Questions Do you use indicators frequently? Are they established in your organization? Have you seen any improvements due to the use of indicators? Are standardized indicators a good idea?