1. Disaster Surveys and Their Validity: Comparing Discrete Distributions Kishore Gawande Texas A&M University Gina Reinhardt Texas A&M University Carol Silva University of Oklahoma Domonic Bearfield Texas A&M University

2. Validity Internal Does the study measure what it’s intending to measure? A  B ? Randomized controlled experiments External Would this study function the same way in a different time, place, or context? Would A  B elsewhere? Representative samples

3. Survey Experiments Internal Validity A  B ? Randomized assignment into control v. treatment groups External Validity Would A  B elsewhere? Representative samples polled Sampling Frame Telephone, internet, dual frame

4. Survey Experiments – Internet Frames Internal Validity A  B ? Randomized assignment into control v. treatment groups is possible External Validity Would A  B elsewhere? Offer pre-established sampling frames Known probabilities, easily comparable to populations of interest So! It’s allllll good. Right?

5. Disasters are… Unplanned disruptions in social and political mechanisms and systems (Quarantelli, Lagadec, and Boin 2006) Inherently social phenomena (Perry 2006) Unpredictable – Planning for their research Funding, access, IRB clearance

6. Disasters and Validity Internal Validity – A  B ? Randomized assignment into control v. treatment (“no disaster” v. “disaster”) groups – Rarely possible, never ethical – Could be hypothetical, but rarely done well (North and Norris 2006) Natural experiment – pre-post comparisons, affected-unaffected comparisons

7. Disasters and Validity External Validity Would A  B elsewhere? Very high exposure to none at all Populations unknowable, unreachable, inaccessible Geographic foci offer a place to begin Victims die or disperse Access is denied or restricted  Convenience samples, selection bias Noncoverage (Brodie et al 2006; Jenkins et al 2009; Kessler et al 2007) Nonresponse (Schlenger & Cohen Silver 2006; Hussain, Weisaeth, & Heir 2009)

8. What Shall We Do? Don’t give up! – Disaster surveys are more prone to problems that plague all surveys Three-fold Solution: – Survey a representative sample using online sampling frames despite widespread displacement of respondents – Assess external validity with a distribution test: the discrete Kolmogorov-Smirnov (KS) test – Include a question from a well-established and respected survey, enabling the design of a survey and quasi-experiment regarding an unpredicted event with the intention of post-validation

9. Why Shall We Do It? 2011 in the US – Hurricane Irene is the 10 th billion-dollar weather/climate event of the year – Total cost pre-Irene: over $33.5 billion, at least 577 deaths 2011 in the World – Earthquake, Tsunami, Nuclear Accident in Japan – Earthquakes/tsunamis in Argentina, Chile, India, Indonesia

10. Why Shall We Do It? Hurricanes Katrina and Rita and Wilma – 2005 – deadliest and costliest hurricane season on US record since 1928 – $168 billion in damages – estimated 1987 deaths So what? – Risk, Sociology, Epidemiology, Public Health – Small samples, convenience, interviews

11. Why Shall We Do It? Advances in survey sampling technology No Surge in disasters No Disasters and their management affect: Public opinion Trust and legitimacy Electoral behavior Structure, health, safety

12. What do we show? Methodological – The discrete Kolmogorov-Smirnov (KS) test is a simple and effective device for validating and analyzing ordered responses typically found in surveys Substantive – Research on unpredicted events does not have to be restricted to small-N or qualitative work – Disaster experiments need not be held to samples of convenience

13. How we’ll show it Disasters and Validity Our Solution Online sampling frame Distribution test: the discrete Kolmogorov-Smirnov (KS) test Question from a well-established and respected survey Our Study Methods: KS test v. Chi-square test Apply our solution to our study Evaluate policy question: Likelihood of returning to hurricane- ravaged area

14. Our Study Hurricanes Katrina and Rita – August and September, 2005 – $150.9 billion – 1952 deaths (Lott et al 2011) – displaced 4 million people (estimated, NOAA 2011) Sample: Hurricane-threatened respondents – Conducted September 2006 – Residents of Counties/Parishes at general risk of hurricane damage – States on Gulf and South Atlantic coast (Texas – North Carolina) – No more than one county/ parish from the coast

15. Our Study Respondents registered in SSI database by county/parish of residence Invitation by email $2.50 for complete survey Entry in $5000 lottery Restriction of Floridians to 1000 Results: 7024 respondents 1576 affected by Katrina/Rita directly 894 evacuated for Katrina (414 displaced) 994 evacuated for Rita (360 displaced)

16. Solution, Step 2: Distribution Test