1. Producing Data and Designing Research Projects Feb 14-15, 2012 PPAL 6200

3. To start…. To start today I just want to briefly point out we are actually going to talk about something not in the book at all, then Chapter 9 before we do Chapter 8. Not sure why the book is arranged the way it is but it seems to make more sense to discuss designs for studies before you discuss sampling.

4. Outline Some Really Important Things Not in the Book: The Research Cycle The Different Designs Research Can Take Sampling

5. Some things not in the book In order to develop a social science project we need to look at some of the steps involved in moving from an idea to a completed project.

6. The Research Development Process or Cycle Stage 1 A Problem or Observation A Question Background Research –Generally a literature of previous research on topics related to your question.

7. Stage 2 A Hypothesis –A statement that something (the explanatory or independent variable[s]) causes or influences variation in something else (the dependent or response variable), and possibly how this is done. To do this you must have a conceptual definition of your variables

8. Stage 3 Operationalizing your variables –How are you going to measure your variables –How are you going to collect those measurements

9. How you go about collecting data depends on the problem and question you began with In many cases the literature review you have conducted will reveal not only what people have found on topics related to the question you have, but also how people have gone about studying these topics as well.

10. Common Research Designs Small “n” vs. Large “n” designs Small “n” –Small “n” comparative Mills’ Methods of Agreement and Disagreement –The Case Study –Small “n” elite Position of individuals –Participant Observer Researcher’s own senses –Key point, in all of these control is logical

11. Large “n” Designs –Survey Research (Opinions) –Econometric (National Accounts and Policies) –Demographic. –Quantitative Content Analysis –All of these can involve either original data collection or the secondary analysis of previously collected data. –Key point, in all of these control is exercised statistically.

12. If you do collect original data… Two things you should be concerned about The validity and reliability of your operationalizations –Pilot Studies The comparability of your work to that of others Make life simpler for yourself, look for questions that have been previously used by other researchers which have been field tested and validated, and to which comparisons of findings can be made.

13. Stage 4 Executing your design

14. Stage 5 Analyzing your data

15. Stage 6 Drawing your conclusions

16. Stage 7 Writing your report

17. Break Time

18. Observation vs. Experiments An Experiment is a special kind of research study that imposes or applies a treatment in order to observe responses so as to determine if the treatment causes a change in response An observational study observes variation in individual variables of interest but does not attempt to manipulate them in order to influence or produce responses

19. In many cases social scientists are interested in, it is probably not ethical to conduct genuine experiments.

20. Subjects, Factors, Treatments The book describes a “study” as “an experiment when we actually do something to people, animals,… in order to observe the response.” Not sure. Another view is that a study is any systematic investigation.

21. Some key words we have to remember –Subject –Explanatory Variables or Factors, or Independent Variables –Response Variables, or Dependent Variables –Treatment (can be a combination of Factors)

22. Experiments Good and Bad The benefit of experiments is that they allow for “control”, we try to manipulate one or some variables (the treatment) so as to produce change in another variable (the response) while holding everything else constant If all the variables that can impact the response are not controlled, the results can be compromised.

23. Three types of Control Physical control Statistical control Logical

24. The Double Blind Randomized Comparative Experiment This is the “gold standard” in research design often used in medical and pharmaceutical research. You are interested if treatment “x” can cure a disease. You draw a random sample of the population of those suffering the disease and someone separate from the researchers randomly assigns individuals to either the experimental or control group. You then administer the treatment to the experimental group but provide either a placebo, or more likely, an already approved treatment to the control group so that they also don’t know if they are receiving the treatment or not. Neither researchers nor subjects know who is in the treatment group and who is in the control group You then compare the results for the two groups

25. The Logic of a Randomized Comparative Experiment Randomized selection and assignment to treatment and control groups mean all variations among the cases ought to be equally represented in each group Therefore, any differences in average response must be due either to the treatment or (as occasionally discovered pure random chance). However, if you use enough subjects random chance ought not to play a role.

26. As we noted before, there is always a chance that results are the product of pure chance. A result that is so strong that it is unlikely to be the product of pure random chance is said to be statistically significant.

27. Matched pairs and other block designs Sometimes the gold standard is not possible but other experimental designs are. Essentially this is like a randomized comparative experiment except that the assignment of individuals is done one pair at a time so randomization is not as thorough.

28. Break Time

29. Producing Data DON’T!!!!!!!MY BEST ADVICE REGARDING THE GENERATION OF DATA SUITABLE FOR STATISTICAL ANALYSIS IS DON’T!!!!!!! HOWEVER, having noted that, you do need to be aware of how people go about collecting the data you are using and the limits that are imposed on your work by these methods

30. Sampling… Some Key Terms –Population (the entire group of individual cases about which we want information) –Sample (a part of the population from which we actually collect information and from which we will draw conclusions about the entire population) –Sampling Design (the method we use to choose a sample from the population) –Inference (the process of generalizing our results from our sample to the wider population)

31. The Ideal Sample A sound sampling design is essential if we are going to produce results that are generalizable from our sample to the entire population In the ideal sample design: 1.Each individual case has an equal probability of being drawn as part of the sample 2.Each characteristic of relevance to the study is present in equal probability in our sample and the population-at-large. ****NOTE: by the end of today’s lesson we will see that this second characteristic is not essential as long as: 1.The first point is realized 2.and the sample we draw is large enough However, for now we will assume it is a good idea to aim for

32. Garbage In… Garbage Out… The truth of the matter is –It is very hard to absolutely achieve both of the ideal characteristics of a sample For some populations it might be impossible such as relatively small groups (think about policy-makers and advisors for example) –It is getting harder all the time to do so –The best we can probably ever hope to do is “approximate” the two characteristics. The closer we approximate the two characteristics The better will be the accuracy of our inferences

33. Sample types Convenience Samples Voluntary Response Samples Purposive Samples –Snowball Samples Simple Random Samples –Not only need to randomly choose the individual cases, but also need to figure out the correct size for the sample so that every possible sample has an equal chance to be chosen

34. Stratified Random Sample –Sometimes you want to go beyond a Simple Random Sample so as to ensure certain characteristics are not left to even the chance of error. This is common in public opinion polling where several Simple Random Samples will be drawn for populations identified along geographic lines.

35. The next slide portrays a multistage stratified random sample

36. BCPrairieON QU Atlantic Randomly Select A set of Postal Sorting Areas for each Region and Randomly Select People in Those PSAs National Stratified Random Sample Greater Vancouver Greater Victoria Other Van. Island The South Interior The North

37. Stratification need not be done on geographic lines (though it often is) Any characteristic can be used as the basis of stratification. What matters is that it is something of importance to your research

38. Caution As the book notes, a sample’s quality is built on more than just the sampling technique We need to have reliable information about the population before we can even begin to select a decent sample Non-responses Response bias And many other factors can also degrade a sample’s quality and the quality of the data we can get out of it.