The Role of the Tukey algorithm in validation procedures for prices data in a consumer price index: the UK experience on this and more general aspects of data editing David Fenwick

Editing of CPI prices data:Central Premise 1. Editing is a non-trivial issue. It can have a systematic numerical impact on measured inflation which can lead to bias 2. Automated editing can improve the quality of the index and increase operational efficiency 3. Both of the above statements apply to the Tukey algorithm Auditing and editing needs to be in “real time” because shop prices can change quickly

Some background on ONS editing procedures – salient points Most prices are collected by handheld computer with the facility for interactive editing in real time Two distinct algorithms operate at HQ to identify outliers amongst price quotes, often operating in parallel –Scrutiny, two tests by reference to “average” price of same/similar items (2k from 100k) The minimum-maximum test The percentage change test –Tukey, in essence a more sophisticated version of scrutiny. This is applied to price quotes not identified by scrutiny as outliers (4k from 100k)

Some background on ONS editing procedures – salient points At the time of study the presumption was that an outlier was incorrect, and therefore declared invalid, unless positively verified by reference to metadata from the price collector or by checking the quote with the shop keeper –All “scrutiny” & half of Tukey outliers were subject to positive verification (most were correct). –The rest, that is those not positively verified were assumed incorrect. –The number explicitly accepted after verification was 100 times the number explicitly rejected (indicates imbalance & potential bias)

Some background on ONS editing procedures – salient points Automated filtering mechanisms avoid manual examination of large numbers of prices over a short time But automated filtering mechanisms need to be supported by “well-informed” manual editing –Especially when there can be unpredictable variations in prices (e.g. seasonal goods, sales) Two main issues for ONS & motivation for the research –The efficiency of its editing procedures –The impact on the accuracy of the RPI/CPI.

The Tukey Algorithm Price quotes are ordered by the corresponding price ratios Highest and lowest 5 per cent are flagged for further investigation and excluded –Price ratios equal to 1 are excluded (i.e. no price change) Arithmetic mean of remaining price ratios used to divide remaining price ratios and their lower/upper trimmed means calculated The upper and lower Tukey limits used to flag those price observations which warrant attention are then calculated as follows: –TU =AM (AMU – AM) –TL = AM – 2.5 (AM – AML) where AML is the lower trimmed mean and AMU is the upper trimmed mean. Tukey maximises use of immediate price history –Can be used for monthly or annual change –AML & AMU can be regularly recalculated

Issue 1: efficiency of editing procedures Considerable overlap between –Interactive editing in field Min max test, metadata (S=sale; R=recovery from sale), logistical checks (R must follow S) –“scrutiny” Not real time & less sophisticated than “scrutiny” but quick to identify extreme outliers & can be run without many body of prices data –Tukey Sophisticated but not interactive More efficient than “scrutiny” which identifies many “outliers” which are valid price quotes “scrutiny” reduces efficiency of Tukey by prior exclusion of “scrutiny” outliers from Tukey –“Scrutiny retained because quick & simple but the presumption of an outlier being “wrong until proven right” challenged

Issue 2: impact on index & potential bias Set of outliers may not necessarily adequately overlap set of “incorrect” prices –Initial investigations showed number of Tukey (& “Scrutiny”) outliers which were incorrect was small. Better to focus on –Extreme outliers –Prices that have not changed for many months (editing ignores) Study undertaken of clothing sub-index of RPI

Clothing: underlying analysis Were clothes prices really lower than 15 years ago?

Clothing: underlying analysis

Deeper sales & shallower recoveries? –But not uniform across clothing

The research questions Is the data editor at HQ more reliable than the price collector? –Automated editing automatically over-riding price quotes for replacement items from “non-comparable” to “comparable” where small price difference –“Scrutiny” can overturn in a few seconds a considered decision by price collector “odd” results- only 10% as many “recoveries” as sales Correct categorisation of replacement item as comparable or non- comparable is important. –Tukey doesn’t test this –“Scrutiny” An examination of original & final indicator codes shows over half of “non-comparable” replacements were reclassified at HQ (a half to “comparable”) & 10% of “C”’s were changed to “N”’s Analysis of price relatives indicate latter might have an undue influence on “Scrutiny” with a cumulative effect on index

Research question: what has been the numerical impact of edited changes in “indicator” codes? ONS records original and final (after all editing) “indicator” codes –Original indicator codes gave higher index

Research question: what has been the numerical impact of edited changes in “indicator” codes?

Research question: is the price collectors judgement better than HQs? Auditor back check of indicator codes –Collectors decisions generally accurate & better then HQs

Further thoughts on Tukey Tukey is robust –The implicit thresholds defined by Tukey are not subject to substantial revision on receipt of new data Tukey could be used more effectively –Explicit parameters can be adjusted to identify extreme outliers –Double application First apply Tukey to initial prices data and then to full set Validate but suspend validation decisions from first application until confirmed by second Tukey cannot be applied to centrally collected prices or centrally calculated indices –Too few quotes –Rely on scrutiny –But “centrals” represent the biggest inherent risk to the index (small number of quotes, large weight)

Conclusions Data editing “filtering” mechanisms –Can increase efficiency. But can exclude “correct” prices do not guarantee a better index Tukey can be undermined –by other editing procedures –By the presumption that an outlier is an invalid price unless subsequently positively validated by editing –By automatic re-coding of non-comparable “replacements” as comparable “replacements” e.g. where the price difference is small and there is no reference to the indicator code The judgement of the price collector is generally to be preferred Centrally collected prices and centrally calculated indices account for 40% of the basket and Tukey doesn’t help these

END OF PRESENTATION