1. Sampling Design and Analysis MTH 494 Ossam Chohan Assistant Professor CIIT Abbottabad

2. LECTURE-5 2

3. 3 Analysis and Estimation Analysis errors include any errors that occur when using wrong analytical tools or when preliminary results are used instead of the final ones. Errors that occur during the publication of the data results are also considered as analysis errors. Estimation errors occur when inappropriate or inaccurate weights are used in the estimation procedure thus introducing errors to the data. They also occur when wrong estimators are selected by the analyst.

4. 4 Reducing non-sampling errors Can be minimised by adopting any of the following approaches: – using an up-to-date and accurate sampling frame. – careful selection of the time the survey is conducted. – planning for follow up of non-respondents. – careful questionnaire design. – providing thorough training and periodic retraining of interviewers and processing staff.

5. 5 Reducing non-sampling errors – cont’d - designing good systems to capture errors that occur during the process of collecting data, sometimes called Data Quality Assurance Systems.

6. 6 Sampling error Refer to the difference between the estimate derived from a sample survey and the 'true' value that would result if a census of the whole population were taken under the same conditions. These are errors that arise because data has been collected from a part, rather than the whole of the population. Because of the above, sampling errors are restricted to sample surveys only unlike non-sampling errors that can occur in both sample surveys and censuses data.

7. 7 Sampling errors – cont’d There are no sampling errors in a census because the calculations are based on the entire population. They are measurable from the sample data in the case of probability sampling. More will be discussed in detail in more advanced modules of the training programme.

8. 8 Factors Affecting Sampling Error It is affected by a number of factors including: a.sample size. In general, larger sample sizes decrease the sampling error, however this decrease is not directly proportional. As a rough rule of the thumb, you need to increase the sample size fourfold to halve the sampling error but bear in mind that non sampling errors are likely to increase with large samples. b.the sampling fraction. this is of lesser influence but as the sample size increases as a fraction of the population, the sampling error should decrease.

9. 9 Factors Affecting Sampling Error – cont’d c.the variability within the population. More variable populations give rise to larger errors as the samples or the estimates calculated from different samples are more likely to have greater variation. The effect of variability within the population can be reduced by the use of stratification that allows explaining some of the variability in the population. d.sample design. An efficient sampling design will help in reducing sampling error.

10. 10 Characteristics of the sampling error generally decreases in magnitude as the sample size increases (but not proportionally). depends on the variability of the characteristic of interest in the population. can be accounted for and reduced by an appropriate sample plan. can be measured and controlled in probability sample surveys.

11. 11 Reducing sampling error  If sampling principles are applied carefully within the constraints of available resources, sampling error can be kept to a minimum.

12. 12 Sources – http://www.nss.gov.au/nss/home.nsf/SurveyD esignDoc/4354A8928428F834CA2571AB002479 CE?OpenDocument http://www.nss.gov.au/nss/home.nsf/SurveyD esignDoc/4354A8928428F834CA2571AB002479 CE?OpenDocument – http://www.statcan.ca/english/edu/power/ch6 /nonsampling/nonsampling.htm http://www.statcan.ca/english/edu/power/ch6 /nonsampling/nonsampling.htm – http://www.statcan.ca/english/edu/power/ch6 /sampling/sampling.htm

13. Selection Bias; Occurs when some part of the target population is not in the sampled population. A good sample will be as free from selection bias as possible.

14. EXAMPLES OF SELECTION BIAS Sample of Convenience; it is based on the elements that are available to participate in a study. It is often biased since the units that are easy to select or are likely to respond are usually not representative of the harder to select or no responding units. Using a sample-selection procedure that, unknown to the investigators, depends on some characteristic associated with the properties of interest. 14

15. EXAMPLES OF SELECTION BIAS cont’d Judgment Sample: the investigator uses her or his judgment to select the specific units to be included in the sample. The choice of the sample is normally intended to be ‘representative’ Misspecifying the target population; e.g when the target population for the polls is the registered voters who participated in previous elections but are used to predict the outcome of the current elections. 15

16. EXAMPLES OF SELECTION BIAS cont’d Under Coverage: failing to include all the target population in the sampling frame Substituting a convenient member of a population fro a designated member who is not readily available. Non Response: failing to obtain adequate number of responses from the chosen sample. Allowing Samples to consist entirely of volunteers, such is the case with radio and television call in polls, and the statistics cannot be trusted. 16

17. WHAT GOOD ARE SAMPLES WITH SELCTION BIAS? Though bias, purposive or judgment samples can provide valuable information particularly in the early stages of the investigation. 17

18. MEASUREMENT BIAS This occurs when the measuring instrument has a tendency to differ from the true value in one direction. As with selection bias it must be considered and minimized at the design stage of the survey. 18

19. Unavoidable Instances of Measurement Bias People sometimes do not tell the truth People do not always understand the questions People Forget People give different answers to different interviewers People may say what they think the interviewer wants to hear or what they think will impress the interviewer 19

20. Unavoidable Instances of Measurement Bias cont’d A particular interviewer may affect the accuracy of the response by misreading questions, recording responses inaccurately, or antagonizing the respondents. Certain words mean different things to different people, e.g. ‘do you own a car? may be answered yes or no depending on the respondent’s interpretation of you ( household or individual)? Question wording and order have a large impact on the responses obtained. 20

21. Bias and variability Bias: Systemtic deviation from the true value Design, Conduct, Analysis, Evaluation Lots of examples on page 49-51 21

22. Bias and variability Larger study does not decrease bias ; Drog X - Placebo -7-4-10 mm Hg -7-4-10 Drog X - Placebo mm Hg Drog X - Placebo -7-4-10 n=40 n=200N=2000 Distribution of sample means: = population mean Population mean bias 22

23. Bias and variability There is a multitude of sources for bias Publication bias Selection bias Exposure bias Detection bias Analysis bias Interpretation bias Positive results tend to be published while negative of inconclusive results tend to not to be published The outcome is correlated with the exposure. As an example, treatments tends to be prescribed to those thought to benefit from them. Can be controlled by randomization Differences in exposure e.g. compliance to treatment could be associated with the outcome, e.g. patents with side effects stops taking their treatment The outcome is observed with different intensity depending no the exposure. Can be controlled by blinding investigators and patients Essentially the I error, but also bias caused by model miss specifications and choice of estimation technique Strong preconceived views can influence how analysis results are interpreted. 23

24. Bias and variability Amount of difference between observations True biological: Temporal: Measurement error: Variation between subject due to biological factors (covariates) including the treatment. Variation over time (and space) Often within subjects. Related to instruments or observers Design, Conduct, Analysis, Evaluation 24

25. QUESTIONNAIRE DESIGN Decide what you want to find out; very important step in the writing of a questionnaire: write down the goals of your survey and be precise, then write or select questions that will elicit answers to the research questions and that will encourage persons in the sample to respond to the questions. 25

26. Questionnaire Design cont’d Always test your questions before taking the survey; ideally tested on a small sample of the members of the target population Keep it simple and clear; questions that seem clear to you as the researcher might not be clear to the respondent being interviewed over the phone or respondent with a different native language 26

27. Questionnaire Design cont’d Use specific questions instead of general ones if possible; Relate your questions to the concept of interest; In some disciplines a standard set of questions has been developed and tested, and these are then used by subsequent researchers. Decide Whether to use Open or Closed Questions Report the Actual Question Asked; Avoid Questions the Prompt or Motivate the Respondent to say what you would like to hear Use Forced Choice, Rather than Agree/Disagree, Questions: Some people will agree with everything 27

28. Questionnaire Design cont’d Ask Only one Concept in Each Question; in particular avoid Double-Barreled Questions Pay Attention to Question-Order effects 28

29. Methods of Data Collection Personal Interviews Telephone Interviews Self-Administered Questionnaires Direct Observation 29

30. Concluding Remarks In real life often sampling is driven by funding – monetary concerns. – It is easy to understand the cost and the sample size but not as easy to understand the importance of proper sampling versus convenience sampling. In real life only a single sample is taken and the difference from the estimate and that of the truth can’t be quantified. – Another reason many people go for quantity. Before collecting data think G.I.G.O. - quality over quantity – Statistical tests – p-values may often be performed/calculated using convenience samples but they truly have no meaning when calculated on a convenience sample Finally, the researchers note that much is easier said than done. That is, to take a proper sample is much easier said than done. 30