Workshop on Price Index Compilation Issues February 23-27, 2015 Imputation of Missing Values, Seasonal Products and Quality Changes Gefinor Rotana Hotel, Beirut, Lebanon

Lecture Outline Introduction Imputation Techniques Treatment of Seasonal Commodities Imputation Techniques for QA

Causes of missing prices Samples not universal coverage Temporarily unavailable supply shortages Seasonally unavailable Permanently unavailable

Current practices and their implications Take no action: samples deteriorate bias may be introduced Ensure the observations are comparable (matched between periods) Adjust weights to account for missing data Carry forward the last price can result in bias particularly during inflationary periods Impute prices for missing observations

Imputation Techniques Estimate the price change of comparable observations in the item category Move the previous price forward

Example 1 Imputing a Missing Month’s Value using the Jevons method

Imputation Techniques Estimate a missing current price from the short-term change in average prices of other observations Mark the estimated price as “imputed” If price is missing for several months chain its imputed prices until a replacement is found. See Example 2

Example 2 Imputing Multiple Month’s Missing Prices

Seasonal Items and Prices “ Hard” Seasonality Items missing from markets during part of year At the start of a new season changes in quality introduction of new products At the end of season discounting “Soft” Seasonality Seasonal price movement of year-round items Varying seasonal patterns from year to year

Treatment of Seasonal Commodities The index must reflect the price swing from:the last price (of old season) to: the first price (of new season) Impute missing prices of out-of-season observations prevents systematic bias Particularly when new products are introduced at the beginning of season

Other techniques for seasonal items Variable weights May not give reliable results Requires detailed monthly data for several years on the quantity of each item sold Rolling average: extrapolate recent 6-months Seasonal factors Month-on-(previous) 12-month figure OK. Do NOT use carry forward

Seasonal Clothing Extreme price movement End of season discounts Multiple selling seasons: fall/winter spring/summer Different varieties for each selling season Do not replace variety once price discounting begins

Seasonal Clothing After selling seasons and variety is no longer available, return the price to the last “normal” price begin imputation in the following period When the selling season returns the next year same product may not appear again select the most comparable replacement product (requires maintaining detailed specifications)

Quality Adjustment (QA) Direct comparisons between old and new item Link (splice) new item into the index Drop observation from index calculation

Alternative QA Techniques Direct Quality Adjustment data collector or analyst knowledge, information from producers, or hedonic regression models Indirect Quality Adjustment with Imputation overlap price available overall mean imputation class mean imputation

Example 3—Direct Quality Adjustment

Example 4—Indirect QA: Overlap Pricing

Example 5—Indirect QA: Overall mean imputation

Example 6—Indirect Q A: Class mean imputation

Effect of QA Imputation on Price Change

Recommendation For Missing Price Observations Always impute prices of missing items Use imputed prices in the next pricing period (as the previous period prices) Even in periods for which there are no price observations in the item group at all.

Recommendation For Missing Price Observations (con’t.) The “carry forward” method Use only when it is fairly certain that the price remains the same. Can generate undue stability in the index.

Recommendation For Imputing Missing Price Observations For temporarily missing products use one of the imputation methods (they are self-correcting) For products permanently missing select a replacement If replacement is comparable, use directly If replacement is noncomparable use overlap price or indirect quality adjustment

Thank you