1 IT system and data validation process in Latvian CPI/HICP Prepared by Oskars Alksnis, Central Statistical Bureau of Latvia EU Twinning Project Forwarding Armenian Statistics Through Twinning AM09/ENP-PCA/TP/04 Component E: Harmonized Consumer Price Index Activity E.3: Development of technical methodologies Yerevan 5-9 September 2011

Data collection and data flow in the production of the CPI/HICP Individually designed paper workbooks, containing case specific information and they are valid for 6 months. Prices are entered into database by price collectors. Price collectors work at home. 1 st steps of the data validation as well as check of data consistency Data transmission to central DB (Riga) via . Data check and validation at central office. Case-by-case approach. No any paper work. In case of questions or found mistakes, message has been sent back to price collector via

Data collection workbook

4 Data validation – 1 st stage Reference month Reference month Data entry Load data received from central database (from Riga) Prepare data for transfer to central database (to Riga) Prepare data for transfer to central database (to Riga) Data consistency check and imputations Printouts City/Town Exit Data for checking and confirmation Data for checking and confirmation All data Application for data entry by price collectors

5 Data validation – 1 st stage (2) Address of the outlet Address of the outlet Name of the outlet Name of the outlet Name of the product Reference quantity COICOP code Order number Predefined description of the product Predefined description of the product Actual description of the product filled in by price collectors Actual description of the product filled in by price collectors Price code (e.g. the same as in previous month, discount price, missing price, replacement) Estimated quality change Observed quantity Observed price Observed price Brief gguidelines for price collector Brief gguidelines for price collector Facility to change the arrangement of the products View of historical data up to 12 months Type of the outlet Detailed description of the outlet (if necessary) List of products observed in selected outlet List of products observed in selected outlet Price recalculated automatically to reference unit Price recalculated automatically to reference unit Remarks of price collectors Main data entry window

6 Data validation – 2 nd stage (1) Reference month Reference year City/Town Calculation and view of average prices Data entry – manual or automated Data preparation for sending them to price collectors Check of data consistency and imputation Data transformation to another database for index calculation View the list of all products with detailed descriptions Printouts General information on the processes, e.g. path to stored databases Exit Application for data checking, validation, primary aggregation and storing. This application is available only to central unit staff members

7 Data validation – 2 nd stage (2) COICOP code / search City/Town View of prices Data from the previous month Data from the previous month Data from the current month Data from the current month Order number Order number Observed quantity Observed price Price according to reference quantity Price according to reference quantity Observation code Observation code Estimated quality change where appropriate Estimated quality change where appropriate Observed quantity Observed quantity Observed price Observed price Price according to reference quantity Price according to reference quantity Price index View of detailed guidelines for the selected product Average prices and index for other towns for selected product Average prices and index for country Possibility to use different data filters Possibility to use different data filters Main window for data validation – price section

8 Data validation – 2 nd stage (3) Application of different data filters: by price code, by index value, by price level, by remarks to price collectors, by major product groups, by price collectors, by administered prices

9 Data validation – 2 nd stage (4) View of the product description section Name, address and type of the outlet Predefined description of the product Actual description in previous month Actual description in current month Remarks written by price collectors Main window for data validation – description section

10 Data validation – 2 nd stage (5) View of history section Name, address and type of the outlet Historical data, including information on the observation code, observed quantity, observed and recalculated price Price in lats Time in months Graphical presentation of historical data Main window for data validation – history section

11 Data validation – 2 nd stage (6) View of the section of remarks Person responsible for data validation can write here any questions or remarks to price collector concerning the quality of the data Main window for data validation – section of the remarks and questions to price collectors

12 Future developments – use of portative electronic devices in price collection First attempt -> European Union, Twinning Project LV/2004/FI/01 Central Statistical Bureau of Latvia (CSB), Strengthening Capacity in Particular Subject Matter Divisions  Introduction of the hand-held computers (HHC) in price collection Starting of the implementation process: end 2005 – beginning 2006 Cooperation with CSB’s IT department – no outsource staff Consultancy by representative of Statistics Finland

13 Basic requirements WINDOWS compatible software Light weight Big enough screen Battery capacity around 4-8 hours Accessibility of modem and possibility to send Keyboard type QWERTY Compatibility with current data processing system as much as possible Availability of data validation rules at least in current level Overall acceptance HHC by price collectors

14 Main steps of the implementation Selected model: HP iPAQ hx2400 compatible with Windows Mobile Software was elaborated by IT department of the CSB Instructions provided by CPI unit experts Testing of the first version of program by central staff members Organization of the workshop for price collectors (May 2006) Experimental price collection by some price collectors in Riga (June 2006 – September 2006) Evaluation of the process

15 Main steps of the implementation (2) Some serious shortcomings were detected and recognized  small and inconvenient keyboard  screen size not sufficiently big  missing some useful functions, e.g. view of historical data  missing some of validation rules  limited possibilities to use interactive tools  description of the products not sufficient  technical limitations of the current device Improvements to the program were made

16 Main steps of the implementation (3) Second test of price collection was made by some price collectors in Riga (beginning of 2007) No all drawbacks were eliminated Average time consumption for price collection larger than before Overall resistance by price collectors against the new price collection method Decision on the abandonment of current model Looking for new possibilities -> new models Economical crisis and significant cut of the resources for CSB as well as resistance of price collectors -> development suspended for a while

17 Main conclusions Staff from the CPI unit and, especially, price collectors must be involved in the process actively from the very beginning of the planning and implementation process If the appropriate model or models have been found, first get some couple of devices not all bulk of devices and test it very carefully Design your applications so to get maximum benefit from the technology therefore facilitate the work of price collectors and therefore expecting overall acceptance of new system by price collectors