Political Science Scope and Methods Introduction to Research Design and The Experimental Method

Introduction to Research Design Scientific method (again)  Theory development What qualifies as a theory? Van Evera’s view  Theory testing Van Evera: Positivist approach  Inductive vs. deductive theorizing

Some Terminology Dependent Variables (DV) and Independent Variables (IV)  Van Evera definition  Alternative conception: We explain particular phenomenon – our DV – as a function of specific explanations – our IVs.  Examples  Strategies

Terminology (continued) Internal vs. External validity  Internal validity: the “real effect.”  External validity: “generalizability.”  Threats to validity Example (internal): School vouchers Example (external): Social psychology Bottom Line (76 pages of Campbell and Stanley later): Be Careful!

Testing theories Van Evera: 2 ways to test theories:  Experimentation  Observation Case studies “Large N” (statistical) analysis

Experiments Experimentation:  Lab experiments Effect of negative advertisements (Ansolabehere) Study of political cognition (Berinsky)  Field experiments Effect of canvassing, telephone calls, and mailing on turnout (Green and Gerber) Effectiveness of “franking” – baby books and ballots (Cover and Brumberg)  Survey Experiments War in Iraq (Berinsky)

Example: Iraq War Please give your best guess to this next question, even if you are not sure of the correct answer. As you know, the United States is currently involved in a war in Iraq. Do you happen to know how many soldiers of the U.S. military have been killed in Iraq since the fighting began in March 2003?

Log (Base 10) of Estimates of U.S. Troop Deaths in Iraq, 2004

Table 3: Predicted Probability of Causality Estimates Pr (Underestimate) Pr (Correct Answer) Pr (Overestimate) Information Low Information High Information Difference Follow News About Iraq? Not At All Closely Very Closely Difference Partisanship Strong Republican Strong Democrat Difference

Effect of Information Treatment on Support for War in Iraq Among Under-Estimators Did The U.S. Make The Right Decision in Using Military Force against Iraq? Estimate War Deaths Corrected Information Right Decision48%44% Wrong Decision52%56% N=252;  2 (1)=0.40 Pr=0.53 Has The Current War in Iraq Been Worth Fighting? Estimate War Deaths Corrected Information Worth Fighting58%53% Not Worth Fighting42%47% N=253;  2 (1)=0.71 Pr=0.40

Among Over-estimators Did The U.S. Make The Right Decision in Using Military Force against Iraq? Estimate War Deaths Corrected Information Right Decision42% Wrong Decision58% N=57;  2 (1)=0.00 Pr=0.95 Has The Current War in Iraq Been Worth Fighting? Estimate War Deaths Corrected Information Worth Fighting42%48% Not Worth Fighting58%52% N=57  2 (1)=0.26 Pr=0.61

The Practice of Experimentation Campbell and Stanley: The hard sell Limitation of experiments  Experimental work as the plutonic ideal Experiments are about control  Payoff in causal inference  Maximize internal validity (if do them correctly)  Random Assignment Note: Random assignment  random selection

Other Concerns Construct validity  Why does the treatment work?  Is the treatment what we say it is?

Experiments vs. Quasi Experiments Experiments: C&S – p.8: If you use random assignment, you don’t need to worry about internal validity Quasi-Experiments: C&S – p. 40,56 – things are not so neat  Specific threats to worry about  Designs that control for all threats to validity might be hard to operationalize

Example Enid wants to investigate the effect of saliency of message on attitude change. From an old Ph.D. she finds a swell communication on the importance of physical sciences in a liberal education. Fortunately for her, Widget University conducts separately - English classes for engineers and liberal arts majors. Within this limitation, however, the university has matched the classes carefully on age, sex composition, socioeconomic background, and College Entrance Board Scores (both verbal and mathematical ability as well as on scores in specific subjects). Enid checks on the dean's records and is happy to find that the classes have indeed been matched to the best possible extent. Enid then delivers the message to the engineers (the salient group) and to the liberal arts students (the non-salient group). The engineers show much more attitude change. Enid concludes that message saliency increases attitude change.