1. RESEARCH DESIGN & SURVEYS

2. TOPICS Sampling issues The Instrument (surveys) Bias
Non experimental research design
Experimental vs. quasi-experimental research designs
Research designs with limited power to assess cause and effect
Research design with more power to assess cause and effect

3. Sampling Issues The Sample The population The sampling Frame
The Universe
Census
True parameters
The sampling Frame
PROBABILISTIC
Random
Stratified
NON PROBABILISTIC
Convenient
Snowballing, etc.
The Sample
Does the sample represent the entire population?
Can we make any inference?

4. Making Inferences
Taking only a sample of data from a population becomes necessary as a feasibility/cost issue.
However, even when we seem to have a full population, treating as such means we also assume there is no error in the data coding.
Hence, some believe we should always use inferential statistical methods when working with surveys or other quantitative studies.

5. Avoiding Biased samples
A sample is valid when it reflects the population from which it is drawn.
The central limit theorem has been proven to show that the larger the sample relative to the population from which it is drawn, the more it will reflect the parameter of the population.
Hence, by definition larger samples are more likely to be accurate than smaller samples.

6. Surveying Methods The sample The instrument (aggregate opinion )
The data gathering
method

7. Bias
Systematic bias occurs, and is a huge problem, when bias is incurred directly from the manner in which a survey is collected.
For example, would a survey evaluating perceptions of Democrats be unbiased if conducted exclusively outside a joint meeting of Christian Evangelicals and the NRA?

8. Bias
Sampling error may also occur, and if minimal not a problem, to the degree that we can not be sure whether we fully captured the parameter of a population.
Nonsampling error results from many other problems of research design, such as poorly worded questions or non-responses of certain subjects. There is always potential error in every empirical project.

9. Data
Data Processing
The raw data
Codifying data

10. Data Analysis and Findings Communication

11. Sample Distributions
Though covered more thoroughly in a few weeks, a sample distribution would be the way a statistic would be distributed if drawn from many samples of a population.
Essentially, this is hypothetical and a best guess of what the population parameters may be if we cannot fully know that in reality.

12. Interviews and Questioning
Environment, demeanor, neutrality, are all necessary to gain samples of data with the least bias.
Focus groups in particular require structure through questions in the hope of excellent qualitative data without leading those who respond to certain answers, although structure may vary: Group interview or Interactive focus group.
The more the researcher needs specific answers to questions, the more they need to structure the session.

13. Questions Questionnaires require much skill in order to avoid error.
Avoiding biases of those collecting the information in types of questions.
Avoid value-laden or biased words that act as queues or triggers for certain political/social groups.
Use clear standard language but also make sure your audience understands the words. Example, asking high schools students whether they are taught civics.

14. Structure of Questions
Open v. Closed. As with focus groups, the more you structure answer options the more you can have answers to specific questions. Open-ended questions are good for a full range of respondent views.

15. Additional Problems to be aware of
Instrumentation: How the instrument (e.g. survey) is conducted. For example, a survey through internet will exclude those without access to such technology.
Selection: Selection effects based on who is and who isn’t included in the experiment or survey. Sampling and instrumentation problems.

16. Additional problems of survey instrumentation or experiments
A rival hypothesis is an alternative explanation that may invalidate what we think as cause and effect. There are a few examples:
History: Events that happen in the outside world during the experiment. Ask opinions about civil liberties after 9/11 compared to before, when 9/11 is not the focus of the study.
Maturation: Acquisition and processing of information by subjects is not constant through time. Could be a real problem for some panel studies.

17. Additional Problems to be aware of
Rivalry between treatment and control group: How aware are the subjects of the experiment. In a survey, this is called the Effect, where respondents try to guess at or please the person asking the questions.
A combination (interaction) between the above threats to validity. (e.g. instrumentation-history)

18. Research Design
I want to study the relationship between level of development and income distribution in developing countries.
I need a hypothesis
I need a strategy (research design) to undertake hypothesis testing
I need to be aware that there is not a perfect strategy (research design) but I need to do the best possible research.

19. The hypothesis
The economic development theory suggests that as countries get richer income distribution worsens but eventually income distribution improves.
HIGH
GAP
DIST.
LOW
LOW
HIGH
LEVEL OF DEVELOPMENT

20. Non experimental research
CROSS SECTIONAL
TREND
LONGITUDINAL
COHORT
PANEL
Source: Sampieri et al. 1998:192

21. NON EXPERIMENTAL RESEARCH
CROSS SECTIONAL: The researcher would take a sample of countries with different levels of development (low, medium, high) and analyze the match between the theory and facts.
This approach is the weakest to determine causality, but strong in generalizing.

22. GINI CONCENTRATION RATIOS
LOW INCOME
GINI
Bangladesh
.375
Sri Lanka
.485
MIDDLE INCOME
Mexico
.523
Brazil
.569
Costa Rica
HIGH INCOME
USA
.369
Sweden
.288
Canada
.338
Japan
.285
GINI=0
TOTAL EQUALITY
GINI =1
TOTAL INEQUALITY

23. NON EXPERIMENTAL RESEARCH
LONGITUDINAL: The researcher would select a country or countries that have move from lower to higher levels of development (eg. USA, Canada, etc.) and determine whether income distribution over time followed the trend suggested by the theory
Causality can be infer but lacks detail about the true causes as well as weak on generalization.

24. LONGITUDINAL
TENDENCY: EVOLUTION OF INCOME DISTRIBUTION OVER TIME IN MEXICO.
COHORT: EVOLUTION OF INCOME DISTRIBUTION OVER TIME OF MEXICAN WOMEN BORN AFTER 1960.
PANEL: EVOLUTION OF INCOME DISTRIBUTION OF THE SAME GROUP OF WOMEN OVER TIME (MARIA, SUSAN, BRENDA, JOAN, GLORIA & ISABEL)

25. CASE (S) STUDY
The researcher will take a case (s) study and analyze in depth what factors (policies, institutions, political systems, culture, etc.) are associated with improving income distribution in a given country.
Why is income distribution better in Chile than in Mexico and Brazil despite having the same level of development?
Improves details and understanding of causes and effect but lacks generalization.

26. Experimental vs. Quasi-experimental Research Designs
Experimental research design: The researcher has control over the experiment in terms of sample selection, treatment, environment, etc.
Experimental designs are typical in psychology, medicine, education, etc.
Quasi-experiments: The researcher does not have control over the experiment, rather the experiment occurs in a “natural” setting.
Quasi-experimental design are typical in economics, sociology, public administration, urban planning, political sciences, etc.

27. 0t = Observation in time t of experimental group
RESEARCH DESIGN
0t = Observation in time t of experimental group
X = Treatment
0c = Control group

28. Research design with limited power
POST-TEST ONLY
X O 1
POST-TEST WITH CONTROL GROUP
X O1 Oc

29. PRE-TEST POST-TEST
O1 X O2
PRE-TEST POST-TEST WITH CONTROL GROUP
Oc Oc

30. Research designs with more causal power
CONTROL WITH MORE OBSERVATION IN THE PRETEST
O1 O2 O3 X O4
Oc Oc Oc Oc

31. PRE-TEST POST-TEST REMOVING THE TREATMENT
O1 X O2 X O3 X 04

32. CHANGES TO LOOK FOR CONVERGENCE-DIVERGENCE
Positive change in the treatment group without change in the control group
Treatment
Control
Treatment
Pre-test
Post-test

33. Divergence Positive increments at a different rate Treatment Control
Post-test
Pre-test

34. Convergence The treatment group catches up with the control group
Pre-test
Postest

35. Cross pattern The treatment group overpass the control group Treatment
Pre-test
Post-test

36. Research design with more power (time series)
Pre-test post-test
O1 O2 O3 X O4 O5 O6
Pre-test post-test with control group
Oc Oc Oc Oc Oc Oc

37. Changes to look for in time series
No effect
Change on the rate or slope
Change on the intercept X
POST-TEST
TIME
PRE-TEST