SAMPLING AND NON-SAMPLING ERRORS ISSUES TO CONSIDER BISHWA NATH TIWARI UNDP-APRC BANGKOK 14 MARCH 2013 First Regional Training Assessing Costs and Benefits of Adaptation: Methods and Data 1

CONTENT OF PRESENTATION Household Survey design and errors Steps for conducting survey Errors Sampling and non-sampling errors Suggestion to minimize those errors – Issues to consider 2

NEED FOR DATA Data is critical for effective planning, design & implementation of adaptation projects BUT there is lack of disaggregated data by: Region, districts, settlements (informal), villages Sex and age groups Caste & ethnic groups Economic groups Vulnerable/disadvantaged groups Lack of updated data, and regular availability of data Lack of quality data (with less errors) 3

SOURCES OF DATA Sources of Data SecondaryNationalinternationalPrimary Household survey Institutional survey Some international sources of data: Internal Energy Agency, Global Footprint Network, Centre for International Earth Science Information Network, Centre for Research on the Epidemiology of Disasters, World Bank World Development Indicators Generally, secondary data lack disaggregation CC is global externality but its impacts are local, therefore need for hh level data 4

STEPS FOR CONDUCTING A HOUSEHOLD SURVEY 1.Define universe –households where the adaptation project is to be implemented 2.Determine sample size 3.Decide sampling technique 4.Prepare sampling frame and sample households 5.Develop questionnaire taking into account the variables affecting the adaptation capacity and pretest the questionnaire 6.Organize field survey – selection and training of enumerators, organization in groups with division of responsibility including supervisory responsibility 7.Check data in the field & minimize non-response 5

 Gay (2005) conducted a survey in coastal town of Dumangas in the Philippines to know if farmers are willing to pay to reduce their vulnerability, and if so, how much do they value a planned adaptation program to climate change.  Using CVM, a WTP survey was conducted among 450 hhs; 391 (87%) were willing to pay for planned adaptation program, 59 were not.  The mean and median WTP for planned adaptation program were PHP34.37 and PHP23.96 per month, respectively.  Factors influencing farmers’ WTP for planned adaptation program were education, farm experience, farm size, knowledge about climate change, land tenure, access to credit, and household income.  Provision of alternative livelihoods and training that are less affected by climate change were the most important planned adaptation projects for farmers  Relocation for people affected by sea level rise was the least preferred option Source: Defiesta, Gay (2010). Social Vulnerability and Willingness to Pay for Adaptation to Climate Change of Farmers in Dumangas, Iloilo.A Dissertation submitted to SEARCA college, the Philippines. EXAMPLE OF HH SURVEY: DUMANGAS FARMERS ARE WILLING TO PAY FOR ADAPTATION PROGRAM 6

EXAMPLE OF HH SURVEY: SOUTH AMERICAN FARMERS ADAPT TO CLIMATE CHANGE BY CHANGING CROPS Estimating a logit model across 949 farmers in 7 countries, Seo and Mendelsohn (2008) found that both temperature and precipitation affect the crops that South American farmers choose. Farmers choose fruits and vegetables in warmer locations and wheat and potatoes in cooler locations. …. Global warming will cause South American farmers to switch away from maize, wheat, and potatoes towards squash, fruits and vegetables. Predictions of the impact of climate change on net revenue reflect not only changes in yields per crop but also crop switching. The paper use data collected in 7 South American Countries. The Household surveys asked detailed questions on farming activities during the one year period of July 2003 to June Seo, S. Niggol and Robert Mendelsohn (2006). An analysis of crop choice: Adapting to climate change in South American farms. E C O L O G I C A L E C O N O M I C S 6 7 ( ) –

ERROR OR BIAS Sampling error Non- sampling error Total error Error due to sampling design and sampling technique All other error over and above the sampling error 8

SAMPLING ERROR n Cost 9

HOW TO MINIMIZE SAMPLING ERRORS – POINTS TO CONSIDER  Sampling error depends on: Sample size Sampling technique Heterogeneity of universe  Suggestions to minimize sampling errors Use of appropriate probability sampling technique Minimize stages of sampling – design effect Determine sample size taking into account the variance of the main indicators/attributes of the universe Prepare updated sampling frame so as to minimize the non-response 10

SOURCES OF NON-SAMPLING ERROR Instrument Definition/specification Defective questionnaire Defective Measurement tools Respondent Strategic bias Under/over- reporting Non-response enumerator processor Lack of training Poor coding and editing of questionnaire Mistake in data entry Programming errors Non- sampling error 11

SOURCES OF NON- SAMPLING ERRORS – EXAMPLES Specification error: Unless all agricultural crops are included in a survey it is difficult to enumerate income of farmers from agriculture Strategic Bias: While piloting questionnaire of NMIS cycle 3, it was found that some respondents were reporting false info about sources of drinking water with a strategy that they will get tube well. Error from enumerator due to lack of training: (i) Use of lead question; (ii) Lack of probing 12

HOW NON-SAMPLING ERRORS CAN OCCUR- EXAMPLES Using data from a survey of more than 9,000 farmers across 11 African countries, Kurukulasuriya et. al (2006) estimates how farm net revenues are affected by climate change compared with current mean temperature. Revenues fall with warming for dry land crops and livestock, whereas revenues rise for irrigated crops. At first, warming has little net aggregate effect as the gains for irrigated crops offset the losses for dry land crops and livestock. ….The final effects depend on changes in precipitation, because revenues from all farm types increase with precipitation. Because irrigated farms are less sensitive to climate, where water is available, irrigation is a practical adaptation to climate change in Africa. Strategic bias (eg, reporting less revenue) from the farmers with irrigated farms can change the conclusion of the study! A specification error of dry land or irrigated crops can change the result of the study! Source: Kurukulasuriya et. al (2006). Will African Agriculture Survive Climate Change? The World Bank Economic Review. 13

HOW TO MINIMIZE NON-SAMPLING ERRORS – POINTS TO CONSIDER  Use of structured questionnaire and FGDs – skip pattern, codes  Use of survey manual with standard definition and specification  Pre-test of questionnaire  Use of mix of instruments – structured and unstructured questionnaire/FGD  A thorough training of enumerators with background knowledge on the subject  Good rapport building with respondents  No lead question; no guess  Use of probing questions – who, what, where, why and how  Request for time if the questionnaire is not completed in one sitting  Interview place – calm and quiet environment  Improved coding of questionnaire  Programming for data entry and double data entry to minimize possible entry errors  Data cleaning and editing 14

THANK YOU 15