School of Public Administration & Policy Dr. Kaifeng Yang 研究设计 : 实验研究的基本问题

问题 研究设计的核心要诣 实验设计的基本过程和环节 实验设计的影响因素

Given the research question, what type of evidence is needed to answer the question (or test the theory) in a convincing way?

Why? Making decisions to ensure validity Planning: how to collect, analyze, and interpret data in order to test hypotheses 1. Who 2. Where 3. What 4. How 5. When – Can results be generalized (external validity)? – Are the relationships true (internal validity)? – Time order if necessary

School typeAcademic Achiv. Child’s ability School type Academic Achiv. Parent resource Home facility School type Academic Achiv. Parent education value Child valuation School type Academic Achiv. Key: Dealing with rival hypotheses (another example)

When you observe X and Y co-vary, the reason may be: xy z 1 xy z 2 xy z 3 xy z 4 xy z 5 xy z 6

Coping with Rival Hypotheses Operation Fright --  Juvenile Delinquency Visit prison…months ….70% not arrested 1. Even without the program, no more than 30% would have been arrested 2. Family background: Volunteered are different 3. Committed crimes but not been caught 4. Only temporary effects 5. Participation labels them as potential criminals

The New Design 1. Two groups: similar characteristics relevant to delinquency; one attend the program, another not. 2. Interview both groups about delinquent activity and family background before the program 3. Interview all families 4. One year after the visit, interview both groups about delinquent activity and changes in family 5. During step 4, also check arrest and conviction records for both groups 6. Two years after the visit, repeat steps 4 and 5 7. Three years after, repeat 4 and 5.

问题 研究设计的核心要诣 实验设计的基本过程和环节 实验设计的影响因素

PretestStimulusPosttest Randomization Or matching Comparability Compare Exp. Control

Staging Experiments Pretesting Subject recruitment Intro to experiment Manipulation of indep variable Measurement of dep variable Debriefing Acquisition of informed consent Random assignment Manipulation check

Key Features of Experimental Approach 1. Manipulation 2. Control To test hypotheses, the experimenter deliberately introduces changes into the environments of subjects and observes or measures the effects of the changes.

Random Assignment vs. Random Sampling Random sampling is a method of drawing a sample of cases, such as the pool of subjects in an experiment. Random assignment is a method of assigning subjects from the pool to experimental conditions.

Experimental Design Experimental Design (Randomized Controlled) Group Random. Obser.1 Treat Obser.2 Comparison Exper.R e O e1 X O e2 O e2 -O e1 Control R c O c1 O c2 O c2 -O c1 Two groups Randomization Treatment manipulation Pre and post-tests on the dependent variable Control of the environment of the experiment

PretestStimulusPosttest Randomization Or matching Comparability Compare Exp. Control

Pre-experimental vs. True Experimental Designs Pre-experimental designs are designs that contain one or more features of true experimental design features, such a comparison group or random assignment. True experimental designs are those designs that meet the basic requirements for an experiment.

Pre-experimental Designs Design 1: The One-Shot Case Study XO Design 2: The One-Group Pretest-Posttest Design O 1 XO 2 Design 3: The Static-Group Comparison XO 1 O 2

True Experimental Designs Design 4: The Pretest-Posttest Control Group Design O 1 X O 2 R O 3 O 4 Design 5: The Posttest Only Control Group Design X O 2 R O 4

True Experimental Designs (2) Design 6: The Solomon Four-Group Design O 1 X O 2 O 3 O 4 R X O 5 O 6 Designs 4-6 are called between-subjects designs because different groups of subjects are compared with one another. Within subjects design: X 1 O 1 X 2 O 2  Each subject acts as his/her own control – that is, O 1 is compared with O 2 for each subject.

Quasi-experimental Designs Quasi-experimental designs lack some features of a true experimental design, “but permits stronger inferences about cause and effect than pre-experimental designs, by means of special design features and supplementary data.” The quasi-experimental design falls into four subdivided categories, including 1) the separate-sample pretest posttest design, 2) Non-equivalent control group designs, 3) interrupted time-series design, and 4) multiple time-series design.

Quasi-experimental Designs 1) Separate-sample pretest-posttest design O 1 X R X O 4 2) Nonequivalent control group designs O 1 X O 2 R O 3 O 4

Quasi-experimental Designs (3) 3) Interrupted time-series design O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8 4) Multiple time-series design O 1 O 2 O 3 O 4 X O 5 O 6 O 7 O 8 O 1 O 2 O 3 O 4 O 5 O 6 O 7 O 8

Experimentation Outside the Laboratory Field experiments: studies that meet all the requirements of a “true” experiment but are conducted in a natural setting. Experimental designs in survey research Unit of analysis other than individuals  Evidence of the versatility of experimentation comes from the incorporation of experimental design methodology into sample surveys and from the use of units of analysis other than individuals.  Because of its greater degree of control, experimentation is the preferred approach for testing causal hypotheses.

问题 研究设计的核心要诣 实验设计的基本过程和环节 实验设计的影响因素

Experiment as a Social Occasion The particular cues in an experimental situation that communicate to subjects what is expected and what the experimenter hopes to find are called demand characteristics. Subjects in experiments often experience anxiety about being evaluated. Subjects come to experiments expecting the possibility that they will be evaluated.  evaluation apprehension. Experimenter’s expectations about how the experiment will turn out.  experimenters may unintentionally communicate to subjects their expectations about how the subjects “should” respond so as to confirm their hypotheses.

Minimizing Bias Due to the Social Nature of Experimentation Several strategies for handling the problems raised by demand characteristics and experimenter bias 1) To ask subjects about their perception of the experimental situation 2) In a cover story, providing the subject with a false hypothesis about the purpose of the experiment. 3) To measure the dependent variable in a setting different from the one in which the independent variable is manipulated. 4) Double-blind technique: the experimenter does not know which condition a subject is in. 5) Reduce the amount of contact between the experimenter and subjects (e.g., using taped recordings)

Research Design Validity Internal Validity Whether, in a given study or research project, the independent variable did indeed cause or lead to changes in the dependent variable External Validity: About Generalizing whether and to what extent the conclusions in your study would hold for other persons (or other units of analyses) in other places and at other times.

Internal Validity IVDV

Example: Imagine that we are studying the effects of a compensatory education program in mathematics for first grade students on a measure of math performance such as a standardized math achievement test selection

Threats to Internal Validity History: events other than the independent Vs (stimulus) occurred and affected the dependents Maturation: change in subjects over time (e.g., older) Testing: act of measuring affect dependents Regression: from extremes (deviations) to averages Mortality: loss of participants Instrumentation: “measurement” or “process” differs Selection: composition of groups

Program (independent Vs) Observation 2 (dependent Vs) subjects    Observation 1    Who…Where…When…What…How

External Validity: Representativeness

External Validity: Generalizability

Next Week