1. 1 Chapter 14 External Validity of Research

2. 2 External Validity Concept of External Validity Structural Component Functional & Conceptual Components Assessing External Validity Lab Research, Natural Setting & External Validity Summary

3. 3 External Validity: Concept Do results generalize “across”: –Other settings –Populations –Times Do results generalize “to”: –Particular setting –Particular population ? Which is more important to –Basic researchers? Across –Applied researchers? To –? Why? Give examples of limits to from your project

4. 4 External validity: concepts Boundary conditions? –Limitations of generalization Situations where the theory doesn’t apply E.g. Newtonian physics in outer space “Reinforcement theory” when behavior is constrained –? What are some boundary conditions in your study? Ecological validity? –Similarity of conditions to the natural setting Substance abuse in half-way house. v. living at home Training offsite v. on the job

5. 5 External Validity (Whitley’s view) Generalizing Across –“generalizability” Generalizing to –“ecological validity”

6. 6 External Validity Components: Structural –Methods, how its carried out Setting, procedures, sample Functional –How it operates; works Similarity of psych process in study and natural setting Conceptual –How important in real world Faking on personality tests used for applicant selection Marihuana use at work

7. 7 External Validity: Components (Vidmar, ’79) Structural –Method –Procedure –Sample –Setting Example: –Does student participation in GS -> task satisfaction? Generalize to other students? To workers in a factory?

8. 8 External Validity Components (con’t) (Vidmar, 79) Functional (primarily ecological validity) –How close do the psychological processes in study match natural setting? –E.g. mock jury v. real jury Conceptual (primarily ecological validity) –How close does the problem in study match the natural setting? –E.g. crowding in lab study = crowding in prison?

9. 9 Structural Components: Generalize findings to other Settings Participant samples Research procedures Times Cultures ? How are the structural components of your study related to external validity?

10. 10 Structural Component Settings (Physical & Social) –Physical (e.g. Wickland) –Reactivity (behavior changes when being observed) –Research attributes (of E) –Co-participant attributes (when they interact) –Ecological validity (mundane realism) What factors in your project could interact with IVs?

11. 11 Structural Component Participant sample factors –Convenience sample availability v. representative –Restricted sampling E.g. Young, white, lesbian female postal carrier –Volunteer participants Would you “devolunteer”? –Person by person situation interactions Personal characteristics may interact with IV –Ecological validity Are college students representative of…?

12. 12 Research Procedure Factors Artificiality Operational Definitions (use multiple op defs) Levels of IV (what is the range of values?) Ecological Validity (Too many college Ss?) Cultural Factors (Cross cultural factors) Time Factors –(how many psychotherapy sessions are optimal?)

13. 13 Functional and Conceptual Components verisimilitude = appearance of truth Functional –Do the psychological processes act the same way in other settings, populations? –E.g. when the stakes are high in jury decisions v. mock juries? Conceptual –E.g. “sub-clinical” depression in college Ss = form of depression in clinical population? (Kazdin, ’97) –Researchers v. policy makers Estimator vars cannot be changed (e.g. level of victim anxiety) Policy vars can (e.g. time between crime and testimony) –Utility of DV E.g. actual prisoner violence v. perceptions of control (theoretical interest)

14. 14 Relationships Among Components Components of structural, functional, conceptual are interrelated –Acute v. chronic manipulations Related to both functional and conceptual –E.g. are long and short term anxiety the same?

15. 15 Assessing External Validity Assessing Generalizability –1. include generalizability factors E.g. moderators such as age, gender, race –2. replicate findings –3. acknowledge limitations of study ? What are some of yours? Assessing Ecological Validity – consider: –Conceptual (policy relevance of IV; utility of DV) –Functional (similar psychological processes?) –Structural (naturalism?)

16. 16 Lab v. natural Settings & Ecological Validity Lab research & Ecological Validity –Defending lab research (Purposes are different) –Lab research Tests causal hypotheses (Berkowitz 7 Donnerstein, ’82) –“ A behavioral phenomenon reliably deonstrated is a genuine phenomenon” Falsify theoretical porpositions (Higgens & Marlatt, ’73) –Alcoholics don’t drink to reduce tension Dissect complex phenomena (Haney, Banks, & Zimbardo, ’73) –Role assignment alone can determine behaviors Discover new phenomena (Henshel, ’80) –Would ecological research have discovered that biofeedback works?

17. 17 Lab v. Natural Settings (con’t) Ecological Validity as Empirical Question –Can be tested –(e.g. meta of meta-analysis shows effect size r=.73) Natural Settings and Generalizability –No guarantee of generalizability for natural setting research Structural v. Functional & Conceptual Verisimilitude –Ecological focuses on structural, ignores functional/conceptual (Locke, ’86) Analog Research –Reproducing natural setting in lab What most of you would like to do with your manipulation Philosophies of Science and Naturalism in Research –A matter of preference! Basic or Applied? –What’s in your wallet?

18. 18 External Validity & Internal Valitidy It’s a trade off, but… –Internal is more important –Must rule out other plausible causes to draw conclustions E.g. which diet is best? Carbs or Fat? Who really knows? ?Which is more important for your project?

19. 19 External Validity: Summary Concept of External Validity Structural Component Functional & Conceptual Components Assessing External Validity Lab Research, Natural Setting & External Validity