RESEARCH DESIGN & SURVEYS

TOPICS Sampling issues The Instrument (surveys) Bias Non experimental research design Experimental vs. quasi-experimental research designs Research designs with limited power to assess cause and effect Research design with more power to assess cause and effect

Sampling Issues The Sample The population The sampling Frame The Universe Census True parameters The sampling Frame PROBABILISTIC Random Stratified NON PROBABILISTIC Convenient Snowballing, etc. The Sample Does the sample represent the entire population? Can we make any inference?

Making Inferences Taking only a sample of data from a population becomes necessary as a feasibility/cost issue. However, even when we seem to have a full population, treating as such means we also assume there is no error in the data coding. Hence, some believe we should always use inferential statistical methods when working with surveys or other quantitative studies.

Avoiding Biased samples A sample is valid when it reflects the population from which it is drawn. The central limit theorem has been proven to show that the larger the sample relative to the population from which it is drawn, the more it will reflect the parameter of the population. Hence, by definition larger samples are more likely to be accurate than smaller samples.

Surveying Methods The sample The instrument (aggregate opinion ) The data gathering method

Bias Systematic bias occurs, and is a huge problem, when bias is incurred directly from the manner in which a survey is collected. For example, would a survey evaluating perceptions of Democrats be unbiased if conducted exclusively outside a joint meeting of Christian Evangelicals and the NRA?

Bias Sampling error may also occur, and if minimal not a problem, to the degree that we can not be sure whether we fully captured the parameter of a population. Nonsampling error results from many other problems of research design, such as poorly worded questions or non-responses of certain subjects. There is always potential error in every empirical project.

Data Data Processing The raw data Codifying data

Data Analysis and Findings Communication

Sample Distributions Though covered more thoroughly in a few weeks, a sample distribution would be the way a statistic would be distributed if drawn from many samples of a population. Essentially, this is hypothetical and a best guess of what the population parameters may be if we cannot fully know that in reality.

Interviews and Questioning Environment, demeanor, neutrality, are all necessary to gain samples of data with the least bias. Focus groups in particular require structure through questions in the hope of excellent qualitative data without leading those who respond to certain answers, although structure may vary: Group interview or Interactive focus group. The more the researcher needs specific answers to questions, the more they need to structure the session.

Questions Questionnaires require much skill in order to avoid error. Avoiding biases of those collecting the information in types of questions. Avoid value-laden or biased words that act as queues or triggers for certain political/social groups. Use clear standard language but also make sure your audience understands the words. Example, asking high schools students whether they are taught civics.

Structure of Questions Open v. Closed. As with focus groups, the more you structure answer options the more you can have answers to specific questions. Open-ended questions are good for a full range of respondent views.

Additional Problems to be aware of Instrumentation: How the instrument (e.g. survey) is conducted. For example, a survey through internet will exclude those without access to such technology. Selection: Selection effects based on who is and who isn’t included in the experiment or survey. Sampling and instrumentation problems.

Additional problems of survey instrumentation or experiments A rival hypothesis is an alternative explanation that may invalidate what we think as cause and effect. There are a few examples: History: Events that happen in the outside world during the experiment. Ask opinions about civil liberties after 9/11 compared to before, when 9/11 is not the focus of the study. Maturation: Acquisition and processing of information by subjects is not constant through time. Could be a real problem for some panel studies.

Additional Problems to be aware of Rivalry between treatment and control group: How aware are the subjects of the experiment. In a survey, this is called the Effect, where respondents try to guess at or please the person asking the questions. A combination (interaction) between the above threats to validity. (e.g. instrumentation-history)

Research Design I want to study the relationship between level of development and income distribution in developing countries. I need a hypothesis I need a strategy (research design) to undertake hypothesis testing I need to be aware that there is not a perfect strategy (research design) but I need to do the best possible research.

The hypothesis The economic development theory suggests that as countries get richer income distribution worsens but eventually income distribution improves. HIGH GAP DIST. LOW LOW HIGH LEVEL OF DEVELOPMENT

Non experimental research CROSS SECTIONAL TREND LONGITUDINAL COHORT PANEL Source: Sampieri et al. 1998:192

NON EXPERIMENTAL RESEARCH CROSS SECTIONAL: The researcher would take a sample of countries with different levels of development (low, medium, high) and analyze the match between the theory and facts. This approach is the weakest to determine causality, but strong in generalizing.

GINI CONCENTRATION RATIOS LOW INCOME GINI Bangladesh .375 Sri Lanka .485 MIDDLE INCOME Mexico .523 Brazil .569 Costa Rica HIGH INCOME USA .369 Sweden .288 Canada .338 Japan .285 GINI=0 TOTAL EQUALITY GINI =1 TOTAL INEQUALITY

NON EXPERIMENTAL RESEARCH LONGITUDINAL: The researcher would select a country or countries that have move from lower to higher levels of development (eg. USA, Canada, etc.) and determine whether income distribution over time followed the trend suggested by the theory Causality can be infer but lacks detail about the true causes as well as weak on generalization.

LONGITUDINAL TENDENCY: EVOLUTION OF INCOME DISTRIBUTION OVER TIME IN MEXICO. COHORT: EVOLUTION OF INCOME DISTRIBUTION OVER TIME OF MEXICAN WOMEN BORN AFTER 1960. PANEL: EVOLUTION OF INCOME DISTRIBUTION OF THE SAME GROUP OF WOMEN OVER TIME (MARIA, SUSAN, BRENDA, JOAN, GLORIA & ISABEL)

CASE (S) STUDY The researcher will take a case (s) study and analyze in depth what factors (policies, institutions, political systems, culture, etc.) are associated with improving income distribution in a given country. Why is income distribution better in Chile than in Mexico and Brazil despite having the same level of development? Improves details and understanding of causes and effect but lacks generalization.

Experimental vs. Quasi-experimental Research Designs Experimental research design: The researcher has control over the experiment in terms of sample selection, treatment, environment, etc. Experimental designs are typical in psychology, medicine, education, etc. Quasi-experiments: The researcher does not have control over the experiment, rather the experiment occurs in a “natural” setting. Quasi-experimental design are typical in economics, sociology, public administration, urban planning, political sciences, etc.

0t = Observation in time t of experimental group RESEARCH DESIGN 0t = Observation in time t of experimental group X = Treatment 0c = Control group

Research design with limited power POST-TEST ONLY X O 1 POST-TEST WITH CONTROL GROUP X O1 Oc

PRE-TEST POST-TEST O1 X O2 PRE-TEST POST-TEST WITH CONTROL GROUP Oc Oc

Research designs with more causal power CONTROL WITH MORE OBSERVATION IN THE PRETEST O1 O2 O3 X O4 Oc Oc Oc Oc

PRE-TEST POST-TEST REMOVING THE TREATMENT O1 X O2 X O3 X 04

CHANGES TO LOOK FOR CONVERGENCE-DIVERGENCE Positive change in the treatment group without change in the control group Treatment Control Treatment Pre-test Post-test

Divergence Positive increments at a different rate Treatment Control Post-test Pre-test

Convergence The treatment group catches up with the control group Pre-test Postest

Cross pattern The treatment group overpass the control group Treatment Pre-test Post-test

Research design with more power (time series) Pre-test post-test O1 O2 O3 X O4 O5 O6 Pre-test post-test with control group Oc Oc Oc Oc Oc Oc

Changes to look for in time series No effect Change on the rate or slope Change on the intercept X POST-TEST TIME PRE-TEST