Designing Surveys the art and science

For any type and style… Soil biota and effects of land use change Effectiveness of extension organisations Market chains for farm produced timber Local knowledge of water management Genetic variation in medicinal trees Livelihood – environment interactions Mechanisms for science influencing policy Participatory assessment of NRM problems

Principles and practice Principles are very widely applicable –you better know what they are! Application depends on possibilities and constraints of the context –know your situation well –be familiar with methods used by others –refer to guides and methods sections of papers BUT… –traditions and common practice can be inefficient or flawed –some of the most innovative and informative research comes from transfer of methods between disciplines

Objectives It all starts here! Infinite variety, but often either: –current state (and change over time) –patterns, associations, differences… But they need to be very clear. Expect to iterate between objectives and study details

Two examples 1.Increased elephant damage reported in some villages. Are elephants moving along usual migration routes? 2.Is striga infestation worse when fields are suffering from soil erosion?

What are the challenges? 1. Elephants2. Striga PopulationFarmland in areas reporting increased animal damage Maize-growing areas of W Kenya UnitVillage2m x 2m quadrat Sampling scheme30 villages selected at random 10 villages (random) 10 fields per village 2 quadrats per field (1/3 and 2/3 of the way across) Measurement toolQuestionnaire for village meeting Visual assessment of damage Striga counts Soil erosion score

Units The items being studied May be several –person, household, village –tree, field, watershed –gene, individual, family, population –farmer, extension officer, project, organisation

Problems Don’t confuse: unit(s) which are determined by objective and those you measure –may need to move information to new levels ‘Ecological fallacy’ – trying to make inferences about individuals from measurements on groups The opposite also fails! ‘Modifiable areal unit problem’ : the relationship between X and Y depends on the scale at which they are measured.

Population The complete set of units you wish to study Determined by the domain of the study – the extent of the problem you want the information to apply to Common problem –poorly defined population –not stated but implied by sampling

Sampling 4 ideas you have to understand Simple random sampling Stratification Hierarchical (multistage) sampling Systematic sampling Note: good practical sampling schemes will probably use a combination of these

Common problems No explicit sampling scheme –bias –subjective results Weak use of stratification –can greatly improve studies to identify relationships

Stratification to estimate relationships Example: survey to look at ‘market integration’ and relation with poverty. Two variables associated with market integration (mi); distance from main road (d), length of time settled (t) mi d t

Random sampling d 1.Most observations clustered around the mean d, hence poor estimate of response

d t 2.d and t negatively correlated – can not estimate the joint response surface well

Solution Divide into strata Response surface ideas to choose sample size in each stratum May require large sample fraction in some strata NB: often felt by researchers to be ‘biased’ – explain carefully! d t

Sample size Should be based on rational analysis –How much variation is there? –How precise do you want the answer to be? Too large – waste of effort Too small – can not meet objectives Software available to help Expect to iterate –make objectives more modest if required sample size too large

Common problems Arbitrary selection of sample size –illegal in other disciplines! ‘Rules’ which have no basis –‘at least 30 farms for farmer survey’ –‘10% sample’ –NB. Sample fraction (almost) never relevant 50 from 5000 (=1%) gives same precision as 50 from (=0.01%)

Too few ‘higher level’ units. –Eg 1: Objective - factors influencing effectiveness of extension. Sample: 200 farmers from 2 different extension projects. –Eg 2: Objective - SOM change after forest conversion Sample: 200 plots around 1 forest

Too few units for many x – variables –Eg: Determine how adoption depends on gender, education, farm size, group membership, income, and land quality. –Don’t expect to learn all that from observing 30 farms! Forgetting about –effect of sampling scheme (clustering  larger overall size needed) –non-sampling errors

Sampling and non-sampling errors Sampling error: –Due to not measuring whole population –Described by statistical measures (eg standard error, confidence interval) –Control by statistical methods Non-sampling error Non-response measurement errors inaccurate sampling frame Coding or data entry errors Operator differences –Manage by good survey practice –Make allowance in sample size calculation –NB: larger not always better!

Measurement Focus on objectives, use conceptual frameframe Only measure what you know how to use There are always alternatives –How can you measure maize yield in a farmer’s field? Check how others have done it –many guides and manuals available Pilot EVERY measurement tool

Factors affecting impact of extension organizations Farmers learning Farmers planting Farmer to farmer dissem. Impact on incomes/ livelihoods Org. charact- eristics Org. resources External Environ- ment Org. engages Farmers sensi- tized Org. strategy Farmers trained Other benefits Impact indicators - quantity and quality Factors affecting impact

And then… Logistics, transport Workflows and timing Materials and equipment Data handling Quality control …

Ethical issues Prior informed consent Sensitive topics Confidentiality IPR Legal requirements Feedback to data providers