1. CountrySTAT Team-I 10-13 November 2014, ECO Secretariat,Teheran

2.  Introduction  Origin of missing data  Nature of missing data  Implemented methodologies  Proposed methodologies  Results  Conclusion

3.  The objective of this presentation is to introduce basics tools to handle missing data in CountrySTAT and FAOSTAT domains. They are based on simple and friendly approach, easy to use.  The CountrySTAT agricultural production domain was used as a basis to develop and test imputation and validation methodologies that could assist in standardisation across the different statistical domains presents at FAO level.

4.  Data are missing for different reasons 1) The value has not been measured (forget...); 2) The value is measured but lost; 3) The value is measured, but considered unusable (outliers, etc.); 4) The value is measured but unavailable.

6.  In a dataset, data can be 1) Missing completely at random (MCAR): when the events that lead to any particular data-item being missing are independent both of observable variables and of unobservable parameters of interest, and occur entirely at random. P(r |Y observed ;Y missing ) = P(r ) 2) Missing at random (MAR): when the missingness is related to a particular variable, but it is not related to the value of the variable that has missing data. P(r |Y observed ;Y missing ) = P(r |Y observed ) 3) Not missing at random (NMAR): when data are not MCAR or MAR P(r |Y observed ;Y missing ) = P(r |Y observed ;Y missing ) 4) Censored and Truncated Data. Data use to be MCAR or MAR

7.  A) Deductive or logical imputation;  B) Mean imputation;  C) Ratio imputation;  D) Regression imputation;  E) Donor imputation (hot-deck, cold-deck, nearest neighbor) ;  F) Multiple imputation : Because it is not deterministic, it is not applicable to officials statistics.

8.  expert judgment  last observations carried forward  linear interpolation  growth-rate benchmarking yield estimation multivariate approach These imputations are based on deductive or logical imputation, ratio imputation and donor imputation. The selected method is based on Regression imputation method. WHY? already applied under development trend smoothing tested but not applied

10. Year20032004200520062007200820092010201120122013 Area135---195160---170190208210205---

11. A linear trend is assumed to exist between the start- and endpoints of gaps in the time series. Let y 0, y 1,..., y t-l denote the data points with values obtained from official sources before the gap and y t+r, y t+r+1,..., y m denote the data points with official values after the gap. The imputed values are calculated as:

12. Year20032004200520062007200820092010201120122013 Area135195 160--- 208210205

15. Year200520062007200820092010201120122013 Area125135141--133125144--160 Production1125200226952200198210012725--2820

17.  Used methods are based on regression imputation and used EM-algorithm : 1)Yield estimation: estimate yield using an arima model; 2)Linear regression: Use a linear regression between P t and A t including Trend; 3)Arima model: Estimate P t and A t using ARIMA model; 4) Spline regression: Estimate P t and A t using spline;

19.  How it is work ?

20.  Compute a yield time series Y t containing missing data:  Y t =P t /A t, where P t is the production and A t is the area harvested at time t;  Use linear interpolation method to obtain starting values;  ARIMA(0,1,1): Y t =Y t-1 + α+ ε t - θ 1 * ε t-1 ;  EM algorithm.  Use Yield estimate to impute Production and Area Harvested.  Where P t and A t are missing, we use last observation carried forward method to impute area harvested.

21.  The model assumes linear relationship between Production and Area Harvested;  P t = Y t *A t  P t = Production in the year t;  A t = Area Harvested in the year t;  Y t = Yield in the year t.  Algorithm:  1) Linear interpolation for Area for starting values;  2) Repeat and update until the convergence of prediction values:  P t = α+ β 1 *Trend + β 2 *A t + ε t (EM-Algorithm to impute P t )  A t = α+ β 1 *Trend + β 2 *P t + ε t (EM-Algorithm to impute A t )

22.  The ARIMA models must be identified  ARIMA(0,1,1): Y t =Y t-1 + α+ ε t - θ 1 * ε t-1 ;  Use relation between Production and Area  Use these variable as time series and Impute using EM- algorithm.  Package mtsdi of R.  Impute using ARIMA model for Pt and At imputation

23.  Form of interpolation where the interpolant is a special type of piecewise polynomial called a spline.  For each interval, we try estimate a polynomial function which fit well data.  Spline interpolation is preferred over polynomial interpolation because the interpolation error can be made small even when using low degree polynomials for the spline.  Package mtsdi of R.  Impute using Spline regression for Pt and At imputation

24.  We use reals data to test proposed methodologies: Yield estimation, Linear Regression, ARIMA, Spline  We add also linear interpolation  Data are from CountrySTAT-Mali website.  Missing data are generated randomly.  Data are from 1984 to 2012. Use real data to test.

25.  Test case: Maize.  Missing data at 10 %.

32.  We perform again these methods on the same dataset at different percentages of missing data.

33. % MissingMethodMinMaxMeanStd.Dev 10 Linear.Int. Yield Linear Reg. ARIMA Spline 0.092 0.107 0.063 0.008 0.098 0.558 0.354 0.308 0.270 0.332 0.262 0.205 0.191 0.136 0.190 0.202 0.102 0.096 0.097 0.079 20 Linear.Int. Yield Linear Reg. ARIMA Spline 0.011 0.061 0.014 0.050 0.034 1.142 0.540 0.758 0.517 0.281 0.249 0.238 0.312 0.231 0.142 0.303 0.126 0.253 0.142 0.076 40 Linear.Int. Yield Linear Reg. ARIMA Spline 0.011 0.003 0.184 0.026 0.013 0.011 0.003 0.184 0.026 0.013 0.198 0.174 0.235 0.182 0.154 0.160 0.098 0.181 0.106 0.096

34.  For the 3 tests cases, relatives errors are less for method of Spline in the most of case, when the percentage of missing data is more than 10%.  The method ARIMA is more adapted when we have less than 10% of missing data in the dataset.  The above tests use only two variables for the same crop (area and production). If the number of missing data exceeds 40%, it will be appropriated to use a third correlated control variable.

35. THANK YOU