1 Data quality and surveys for SEEAW Training Course on Water Accounting Amman, Jordan March 2008 Michael Vardon United Nations Statistics Division

Outline Data qualityData quality Data sourcesData sources Surveys optionsSurveys options SurveysSurveys Examples of surveys from AustraliaExamples of surveys from Australia

Data quality RelevanceRelevance AccuracyAccuracy TimelinessTimeliness AccessibilityAccessibility InterpretabilityInterpretability CoherenceCoherence Statistics Canada – Quality Guidelines 4 th Edition XIE03001.pdf XIE03001.pdf

Relevance “The relevance of statistical information reflects the degree to which it meets the real needs of clients. It is concerned with whether the available information sheds light on the issues that are important to users. Assessing relevance is subjective and depends upon the varying needs of users. The Agency’s challenge is to weigh and balance the conflicting needs of current and potential users to produce a program that goes as far as possible in satisfying the most important needs within given resource constraints.” Statistics Canada – Quality Guidelines 4 th Edition 2003 Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 6

Accuracy “The accuracy of statistical information is the degree to which the information correctly describes the phenomena it was designed to measure. It is usually characterized in terms of error in statistical estimates and is traditionally decomposed into bias (systematic error) and variance (random error) components. It may also be described in terms of the major sources of error that potentially cause inaccuracy (e.g., coverage, sampling, non response, response).” Statistics Canada – Quality Guidelines 4 th Edition 2003 Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 6-7

Timeliness “The timeliness of statistical information refers to the delay between the reference point (or the end of the reference period) to which the information pertains, and the date on which the information becomes available. It is typically involved in a trade-off against accuracy. The timeliness of information will influence its relevance.” Statistics Canada – Quality Guidelines 4 th Edition 2003 Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Accessibility “The accessibility of statistical information refers to the ease with which it can be obtained from the Agency. This includes the ease with which the existence of information can be ascertained, as well as the suitability of the form or medium through which the information can be accessed. The cost of the information may also be an aspect of accessibility for some users.” Statistics Canada – Quality Guidelines 4 th Edition 2003 Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Interpretability “The interpretability of statistical information reflects the availability of the supplementary information and metadata necessary to interpret and utilize it appropriately. This information normally includes the underlying concepts, variables and classifications used, the methodology of data collection and processing, and indications or measures of the accuracy of the statistical information.” Statistics Canada – Quality Guidelines 4 th Edition 2003 Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Coherence “The coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytic framework and over time. The use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys. Coherence does not necessarily imply full numerical consistency.” Statistics Canada – Quality Guidelines 4 th Edition 2003 Statistics Canada – Quality Guidelines 4 th Edition 2003 p. 7

Data sources AdministrativeAdministrative e.g. Licensing data basese.g. Licensing data bases Company reportsCompany reports Many companies have reports that include information on the environment (e.g. water and energy use, pollution control measures)Many companies have reports that include information on the environment (e.g. water and energy use, pollution control measures) Academic sourcesAcademic sources Can be government or non-governmentCan be government or non-government SurveysSurveys

SurveysData Physical dataPhysical data Monetary dataMonetary data Survey options Adding questions to existing surveysAdding questions to existing surveys Running specialized surveysRunning specialized surveys Survey methodology Mail-out/mail-back, telephone interview, personal interviewMail-out/mail-back, telephone interview, personal interview Mail-out/mail-back the cheapest and hence most often usedMail-out/mail-back the cheapest and hence most often used Regardless of methods it is critical to test forms in the field prior to the survey.Regardless of methods it is critical to test forms in the field prior to the survey. Sample designSample design

Survey testing regime Good questionnaire design will help minimize response biases. Forms should attractive and easy to understand and fill in.Forms should attractive and easy to understand and fill in. Develop a draft of form based on data requirements and knowledge of survey respondents, using terms and concepts familiar to survey designers and survey respondents. Test on a small number via personal interviews (10)Develop a draft of form based on data requirements and knowledge of survey respondents, using terms and concepts familiar to survey designers and survey respondents. Test on a small number via personal interviews (10) Develop 2 nd draft of form and test on a larger number of respondents (30-60) via personal interviews and mail-out/mail-back (if this is the method to be used)Develop 2 nd draft of form and test on a larger number of respondents (30-60) via personal interviews and mail-out/mail-back (if this is the method to be used)

Specialized Surveys Advantage Can collect a range of data need for the accounts as well as additional data to aid interpretation and understandingCan collect a range of data need for the accounts as well as additional data to aid interpretation and understanding Can select sample to represent total water use (not total economic activity)Can select sample to represent total water use (not total economic activity)Disadvantage Costly to develop and runCostly to develop and run

Adding questions to existing surveys Advantage Direct link to the other data collected in the surveyDirect link to the other data collected in the survey Cheaper than running specialized surveysCheaper than running specialized surveysDisadvantage Financial officers or business accountants who typically fill in business survey forms may not be familiar with environmental dataFinancial officers or business accountants who typically fill in business survey forms may not be familiar with environmental data Sample selection may not be idealSample selection may not be ideal

Sample selection may not be ideal For example, in business surveys sample may be selected to estimate total economic activity, not total water use or supply As such large water using industries (food and beverage manufacturing, metal manufacturing, paper manufacturing, electricity generation) may be under-represented in sample design. This problem can be overcome by adding sample in these industries

Example questionnaires from Australia Specialized surveys Water Supply and Sewerage SurveyWater Supply and Sewerage Survey Water Survey (supplement to Agricultural Survey)Water Survey (supplement to Agricultural Survey) Adding questions to existing surveys Mining SurveyMining Survey Manufacturing SurveyManufacturing Survey Agricultural CensusAgricultural Census All surveys mail-out /mail-back

ABS Surveys – Water Supply and Sewerage Industries Survey

ABS Surveys – Water Survey, Agricultural

ABS Surveys – Agricultural Census

ABS Surveys – Economic Activity Survey (Mining)

ABS Surveys – Economic Activity Survey (Manufacturing)

Intensive follow-up, 1 Survey forms will often be filled in by a financial officer or business accountant who may not have easy access to the data requiredSurvey forms will often be filled in by a financial officer or business accountant who may not have easy access to the data required Intensive follow-up required and should be targeted at large water using industries. For example:Intensive follow-up required and should be targeted at large water using industries. For example: Pulp and paper manufacturing,Pulp and paper manufacturing, Food and beverage manufacturingFood and beverage manufacturing Metals manufacturingMetals manufacturing Electricity generationElectricity generation

Intensive follow-up, 2 Steps for intensive follow-up in Australia 1 st Reminder letter1 st Reminder letter 2 nd Reminder letter2 nd Reminder letter Phone call - important to carefully train staff. Also it may be possible to collect key data items over the phonePhone call - important to carefully train staff. Also it may be possible to collect key data items over the phone Final reminderFinal reminder Notice of Direction (fill in the form or be summoned to court)Notice of Direction (fill in the form or be summoned to court)

Data checking / input editing Identification of outliers or improbable (that is very large or very small numbers) are identified for each industry (or region).Identification of outliers or improbable (that is very large or very small numbers) are identified for each industry (or region). Double check very large numbers with the data supplier (often units are wrongly reported (e.g. m 3 reported not 1,000 m 3 or acres not hectares)Double check very large numbers with the data supplier (often units are wrongly reported (e.g. m 3 reported not 1,000 m 3 or acres not hectares) Check blank cells – they may be zeroCheck blank cells – they may be zero

Contact details Michael Vardon Adviser on Environmental-Economic Accounting United Nations Statistics Division New York USA Room DC Phone: Fax: