1. Bad assumptions or bad luck: Why natural hazard maps (forecasts, warnings, etc…) often fail and what to do about it Seth Stein, Northwestern University Robert Geller, University of Tokyo Mian Liu, University of Missouri Tohoku, Japan March 11, 2011 M 9.1 NY Times CNN

2. Challenges Mitigation is insurance against possible disaster, with present costs & potential benefits Use science, engineering & social science to help society develop strategies that make sense relative to alternative uses of resources (steel in schools versus hiring teachers) Process hampered by limited scientific knowledge about fundamental processes & thus future events (L’aquila earthquake) The earth is very complicated and regularly reminds us of the need for humility in the face of the complexities of nature “Whenever I hear ‘everybody knows that’ I wonder how do they know that, and is it true” (D. Jackson)

3. "NASA owes it to the citizens from whom it asks support to be frank, honest, and informative, so these citizens can make the wisest decisions for the use of their limited resources.” Richard Feynman's (1988) report after the loss of the space shuttle Challenger

4. Tools in preparing for natural disasters include - Long term forecasts: 1-2500 yr (earthquakes), 100 yr (climate change), 1-10 yr (hurricane, volcano) - Short term predictions: days (hurricanes), days to months (volcano), hours (tornado) - Real time warnings: (hours to minutes) tsunami, earthquake shaking, hurricane, tornado, flood Sometimes these work, sometimes they fail

5. FAILURES False negative - unpredicted hazard Loss of life & property False positive - overpredicted hazard Wasted resources, public loses confidence Authorities typically ignore, deny, excuse, or minimize failure More useful to analyze failures to improve future performance

6. “We got a hell of a beating. We got run out of Burma and it is humiliating as hell. I think we ought to find out what caused it, go back, and retake it." General Joseph Stilwell U.S. Army, WWII I am not discouraged, because every wrong attempt discarded is another step forward. Thomas Edison

7. 2008: Hurricane Ike predicted to hit Miami

8. Ike’s actual track

9. Ike predicted to bring certain death

10. Actual deaths: < 50 of 40,000 Error 800x

11. If it had been a weekday, Major cost

12. K. Emanuel CNN 8/26/11 NYT 8/28/11

13. Economic loss ? What if weekday?

14. Science News 6/15/91 The local economy collapsed, said Glenn Thompson, Mammoth Lakes' town manager. Housing prices fell 40 percent overnight. In the next few years, dozens of businesses closed, new shopping centers stood empty and townspeople left to seek jobs elsewhere. (NYT 9/11/90)

15. Predicted disaster probabilities are often very inaccurate P(sinking) = 0 P(loss) = 1/100,000

16. Systematic errors often exceed measurement errors Uncertainties are hard to assess and generally underestimated Underestimated uncertainty and bias (bandwagon effect) in measured speed of light 1875- 1960

17. Number of human chromosome pairs 1921-1955: 24 Now: 23

18. CDC reported "strong possibility" of epidemic. HEW thought "chances seem to be 1 in 2” and “virus will kill one million Americans in 1976." President Ford launched program to vaccinate entire population despite critics’ reservations 40 million vaccinated at cost of millions of dollars before program suspended due to reactions to vaccine About 500 people had serious reactions and 25 died, compared to one person who died from swine flu NEGLECTING UNCERTAINTY OVERESTIMATES HAZARD 1976 SWINE FLU “APORKALPSE”

19. Much ado made that on January 1, 2000 computer systems would fail, because dates used only two digits U.S. & other governments established major programs Estimated $300 billion spent on preparations HAZARD OVERESTIMATED: Y2K Few major problems occurred, even among businesses and countries who made little or no preparation

20. Japan seemed ideal for hazard mapping Fast moving (80 mm/yr ) & seismically very active plate boundary with good instrumentation & long seismic history But: 2011 M 9.1 Tohoku, 1995 Kobe M 7.3 & others in areas mapped as low hazard In contrast: map assumed high hazard in Tokai “gap” Geller 2011

21. Hazard maps fail because of - bad physics (incorrect description of earthquake processes) -bad assumptions (mapmakers’ choice of poorly known parameters) - bad data (lacking, incomplete, or underappreciated) - bad luck (low probability events) and combinations of these

22. Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2 (Headquarters for Earthquake Research Promotion) Off Sanriku-oki North ~M8 0.2 to 10% Off Sanriku-oki Central ~M7.7 80 to 90% Off Fukushima ~M7.4 7% Off Ibaraki ~M6.7 – M7.2 90% Detailed model of segments with 30 year probabilities Sanriku to Boso M8.2 (plate boundary) 20% Sanriku to Boso M8.2 (Intraplate) 4-7% Off Miyagi ~M7.5 > 90% J. Mori Assumption: No M > 8.2

23. Giant earthquake broke all of the segments 2011 Tohoku Earthquake 450 km long fault, M 9.1 (Aftershock map from USGS) J. Mori Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2 (Headquarters for Earthquake Research Promotion)

24. Tsunami runup approximately twice fault slip (Plafker, Okal & Synolakis 2004) M9 generates much larger tsunami Planning assumed maximum magnitude 8 Seawalls 5-10 m high CNN NYTStein & Okal, 2011

25. http://www.coastal.jp/tsunami2011/index.ph p?FrontPage http://www.geol.tsukuba.ac.jp/~ yagi-y/EQ/Tohoku/ Tsunami radiates energy perpendicular to fault Thus largest landward of highest slip patches

26. Didn’t consider historical record of large tsunamis NYT 4/20/11

27. Lack of M9s in record seemed consistent with model that M9s only occur where lithosphere younger than 80 Myr subducts faster than 50 mm/yr (Ruff and Kanamori, 1980) Disproved by Sumatra 2004 M9.3 and dataset reanalysis (Stein & Okal, 2007) Short record at most SZs didn’t include larger multisegment ruptures Stein & Okal, 2011

28. Most plate motion assumed aseismic, as in Kuriles: Same plate pair, further north - expected only M8 Since 1952 largest thrust earthquakes M8 with average slip 2-3 m. Previous earthquake sequence ~100 years earlier, so inferred seismic slip rate 2-3 cm/yr This is 1/3 of plate motion predicted from relative motion models, so the remaining 2/3 was assumed to occur aseismically. Kanamori, 1977

29. Nature, 2003 By 2003, recognize larger (M9?), rarer, multisegment ruptures that account for much of the proposed aseismic slip 80 mm/yr convergence: M 8: 2.5 m / 100 yr = 25 mm/yr + M 9: 10 m / 500 yr = 20 mm/yr

30. NY Times 3/21/11

31. Scientists will “be able to predict earthquakes in five years.” Louis Pakiser U.S. Geological Survey, 1971 “We have the technology to develop a reliable prediction system already in hand.” Alan Cranston, U.S. senator, 1973 “The age of earthquake prediction is upon us” U.S. Geological Survey, 1975 1970’s optimism Similar in Japan, China, USSR

32. Failed prediction: 1975 Palmdale Bulge USGS director McKelvey stated that “a great earthquake” would occur “in the area... possibly within the next decade” that might cause up to 12,000 deaths, 48,000 serious injuries, 40,000 damaged buildings, and up to $25 billion in damage. Systematic uncertainty of leveling data underestimated SAF

33. In 1985, USGS predicted next with 95% confidence by 1993 Occurred in 2004 (16 years late) M 5-6 earthquakes about every 22 years: 1857, 1881, 1901, 1922, 1934, and 1966 Discounting misfit of 1934 quake predicted higher confidence Failed prediction: Parkfield

34. 2001 hazard map http://www.oas.org/cdmp/document/seismap/haiti_dr.htm 2010 M7 earthquake shaking much greater than predicted for next 500 years 6 mm/yr fault motion

35. Accuracy of hazard map prediction depends on accuracy of answers assumed to hierarchy of four basic questions Where will large earthquakes occur? When will they occur? How large will they be? How strong will their shaking be? Uncertainty & map failure result because these are often poorly known

36. Where do we expect earthquakes? Can use Earthquake history Plate motions Geology GPS Often, we have to chose which to use Different choices lead to different predicted hazards

37. GSHAP 1999 NUVEL-1 Argus, Gordon, DeMets & Stein, 1989 Swafford & Stein, 2007 Slow plate boundary Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

38. 2004 2003 Swafford & Stein, 2007 GSHAP 1999 NUVEL-1 Argus, Gordon, DeMets & Stein, 1989 M 6.4 M 6.3 Slow plate boundary Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

39. USGS 2008 Wenchuan earthquake (Mw 7.9) was not expected: map showed low hazard

40. Hazard map - assumed steady state - relied on lack of recent seismicity Didn’t use GPS data showing 1-2 mm/yr Earthquakes prior to the 2008 Wenchuan event Aftershocks of the Wenchuan event delineating the rupture zone M. Liu

41. Long record needed to see real hazard Swafford & Stein, 2007 1933 M 7.3 1929 M 7.2

42. “Our glacial loading model suggests that earthquakes may occur anywhere along the rifted margin which has been glaciated.” Stein et al., 1979 1985 Concentrated hazard bull's-eyes at historic earthquake sites 2005 Diffuse hazard along margin GSC Map depends greatly on assumptions & thus has large uncertainty

43. Peak Ground Acceleration 10% probability of exceedance in 50 years (once in 500 yr) GSHAP (1999) Present Study HUNGARY: ALTERNATIVE HAZARD MAPS Concentrated hazard inferred from historic seismicity alone Diffuse hazard inferred incorporating geology Toth et al., 2004

44. Plate Boundary Earthquakes Major fault loaded rapidly at constant rate Earthquakes spatially focused & temporally quasi-periodic Past is fair predictor Intraplate Earthquakes Tectonic loading collectively accommodated by a complex system of interacting faults Loading rate on a given fault is slow & may not be constant Earthquakes can cluster on a fault for a while then shift Past can be poor predictor Plate A Plate B Earthquakes at different time Stein, Liu & Wang 2009

45. New Madrid 1991: because paleoseismology shows large events in 900 & 1450 AD, like those of 1811-12 GPS studies started, expecting to find strain accumulating consistent with large (~M7) events ~500 years apart Surprising result

46. Science, April 1999 Little or no motion! Seismicity migrates Recent cluster transient, possibly ending Hazard overestimated

47. “Large continental interior earthquakes reactivate ancient faults … geological studies indicate that earthquakes on these faults tend to be temporally clustered and that recurrence intervals are on the order of tens of thousands of years or more.” (Crone et al., 2003) Similar behavior in other continental interiors

48. during the period prior to the period instrumental events Earthquakes in North China Large events often pop up where there was little seismicity! Ordos Plateau Shanxi Graben Bohai Bay Beijing 1303 Hongtong M 8.0 Liu, Stein & Wang 2011 Weihi rift

49. during the period prior to the period instrumental events Earthquakes in North China Large events often pop up where there was little seismicity! Ordos Plateau Shanxi Graben Bohai Bay Beijing 1556 Huaxian M 8.3 Weihi rift Liu, Stein & Wang 2011

50. during the period prior to the period instrumental events Earthquakes in North China Large events often pop up where there was little seismicity! Ordos Plateau Shanxi Graben Bohai Bay Beijing 1668 Tancheng M 8.5 Weihi rift Liu, Stein & Wang 2011

51. during the period prior to the period instrumental events Earthquakes in North China Large events often pop up where there was little seismicity! Ordos Plateau Shanxi Graben Bohai Bay Beijing 1679 Sanhe M 8.0 Weihi rift Liu, Stein & Wang 2011

52. during the period prior to the period instrumental events Earthquakes in North China Large events often pop up where there was little seismicity! Ordos Plateau Shanxi Graben Bohai Bay Beijing 1966 Xingtai M 7.2 1976 Tangshan M 7.8 1975 Haicheng M 7.3 Weihi rift Liu, Stein & Wang 2011

53. No large (M>7) events ruptured the same fault segment twice in past 2000 years In past 200 years, quakes migrated from Shanxi Graben to N. China Plain Historical Instrumental Shanxi Graben Weihi rift

54. Maps are like ‘Whack-a-mole’ - you wait for the mole to come up where it went down, but it’s likely to pop up somewhere else.

55. When do we expect earthquakes? When we have a long history, we can estimate the average recurrence time - but there’s a lot of scatter When we have a short history, we estimate the recurrence time of large earthquakes from small ones, but this can be biased In either case, we have to assume either that the probability of large earthquakes stays constant with time, or that it changes Different choices lead to different predicted hazards

56. EARTHQUAKE RECURRENCE IS HIGHLY VARIABLE M>7 mean 132 yr  105 yr Estimated probability in 30 yrs 7-51% Sieh et al., 1989 Extend earthquake history with paleoseismology

57. When we have a long history, we can estimate the average recurrence time - but there’s a lot of scatter Mean 132  105Mean 180  72 We can describe these using various distributions - Gaussian, log-normal, Poisson but it’s not clear that one is better than another

58. Gutenberg-Richter relationship log 10 N = a -b M N = number of earthquakes occurring ≥ M a = activity rate (y-intercept) b = slope M = Magnitude When we have a short history, we estimate the size & recurrence time of large earthquakes from small ones, but this can be biased

59. GUTENBERG-RICHTER RELATIONSHIP: INDIVIDUAL FAULTS Wasatch Basel, Switzerland paleoseismic data instrumental data Youngs & Coppersmith, 1985 Meghraoui et al., 2001 paleoseismic data historical data Largest events deviate in either direction, often when different data mismatch When more frequent than expected termed characteristic earthquakes. Alternative are uncharacteristic earthquakes These - at least in some cases - are artifacts of short history that overpredict or underpredict hazard Characteristic Uncharacteristic

60. POSSIBLE BIASES IN ESTIMATING THE MAGNITUDE AND RECURRENCE TIME OF LARGE EARTHQUAKES FROM THE RATE OF SMALL ONES Undersampling: record comparable to or shorter than mean recurrence - Usually find too-short recurrence time. Can also miss largest events Direct paleoseismic study: Magnitude overestimated, recurrence underestimated Events missed, recurrence overestimated Earthquake Rate Stein & Newman, 2004 CHARACTERISTIC UNCHARACTERISTIC

61. SIMULATIONS Short history: often miss largest earthquake or find a too-short recurrence time 10,000 synthetic earthquake histories for G-R relation with slope b=1 Gaussian recurrence times for M> 5, 6, 7 Various history lengths given in terms of T av, mean recurrence for M>7 Stein & Newman, 2004

62. Long history: often still find too-short or too-long recurrence time Stein & Newman, 2004

63. RESULTS VARY WITH AREA SAMPLED Stein et al., 2005 Increasing area around main fault adds more small earthquakes Characteristic earthquakes on Wasatch fault (Chang and Smith, 2002), but not in entire Wasatch front (data from Pechmann and Arabasz, 1995)

64. Time dependent predicts lower until ~2/3 mean recurrence Results depend on both model choice & assumed mean recurrence Assumed probability of large earthquake & thus hazard depend on recurrence model & position in earthquake cycle Hebden & Stein, 2008 Not clear which model works best where

65. CHARLESTON 2% in 50 yr (1/2500 yr) Hebden & Stein, 2008 At present, time dependent predicts ~50% lower hazard Still less in 2250

66. California Time- dependant probabilities Increased on southern San Andreas

67. What will happen in large earthquakes? Major unknowns are magnitude of the earthquake and the ground shaking it will produce Tradeoff between these two parameters Different choices lead to different predicted hazards

68. EFFECTS OF ASSUMED GROUND MOTION MODEL Effect as large as one magnitude unit Frankel model predicts significantly greater shaking for M >7 Frankel M 7 similar to other models’ M 8 Newman et al., 2001

69. PREDICTED HAZARD DEPENDS GREATLY ON - Assumed maximum magnitude of largest events -Assumed ground motion model -Neither are known since large earthquakes rare 180% 275%

70. What to do Continue research on fundamental scientific questions Realistically assess uncertainties stemming from current limited knowledge and present them candidly to allow users to decide how much credence to place in maps Develop methods to objectively test hazard maps and thus guide future improvements

71. Global warming forecasts present uncertainties by showing factor of 3 range of model predictions The AOGCMs cannot sample the full range of possible warming, in particular because they do not include uncertainties in the carbon cycle. In addition to the range derived directly from the AR4 multi-model ensemble, Figure 10.29 depicts additional uncertainty estimates obtained from published probabilistic methods using different types of models and observational constraints: the MAGICC SCM and the BERN2.5CC coupled climate-carbon cycle EMIC tuned to different climate sensitivities and carbon cycle settings, and the C 4 MIP coupled climate-carbon cycle models. Based on these results, the future increase in global mean temperature is likely to fall within –40 to +60% of the multi-model AOGCM mean warming simulated for each scenario. This range results from an expert judgement of the multiple lines of evidence presented in Figure 10.29, and assumes that the models approximately capture the range of uncertainties in the carbon cycle. The range is well constrained at the lower bound since climate sensitivity is better constrained at the low end (see Box 10.2), and carbon cycle uncertainty only weakly affects the lower bound. The upper bound is less certain as there is more variation across the different models and methods, partly because carbon cycle feedback uncertainties are greater with larger warming.Figure 10.29 Box 10.2 IPCC 2007 Warming by 2099

72. In addition to comparing maps, comparing model predictions shows the large uncertainties resulting from different assumptions Shows contributions to logic tree before subjective weighting

73. Testing analogy: evidence-based medicine objectively evaluates widely used treatments Although more than 650,000 arthroscopic knee surgeries at a cost of roughly $5,000 each were being performed each year, a controlled experiment showed that "the outcomes were no better than a placebo procedure."

74. Bad luck or bad map? One test is to compare maximum acceleration observed over the years to that predicted by both map and null hypotheses. A simple null hypothesis is regionally uniformly distributed hazard. Japanese map seems to be doing worse than this null hypothesis. Geller 2011 Need objective criteria to test maps by comparison to what happened after they were published.

75. Some testing challenges 1)Short time record: can in some cases be worked around. For example, North China record probably has almost or all M7s in 2000 years. Paleoseismology can go back even further, with higher probability of missing some. 2)Subjective nature of hazard mapping, resulting from need to chose faults, maximum magnitude, recurrence model, and ground motion model. This precludes the traditional method of developing a model from the first part of a time series and testing how well it does in the later part. That works if the model is "automatically" generated by some rules (e.g. least squares, etc). In the earthquake case, this can't be done easily because we know what happens in the later part of the series.

76. 3) Biases due to new maps made after a large earthquake that earlier maps missed. Frankel et al, 2010 Before 2010 Haiti M7After 2010 Haiti M7 4X

77. A posteriori changes to a model are "Texas sharpshooting:” shoot at the barn and then draw circles around the bullet holes.

78. 4) Overparameterized model (overfit data): Given a trend with scatter, fitting a higher order polynomial can give a better fit to the past data but a worse fit to future data Analogously, a seismic hazard map fit to details of past earthquakes could be a worse predictor of future ones than a less detailed map How much detail is useful? Linear fit Quadratic fit

79. Summary - Hazard maps depend dramatically on unknown and difficult-to-assess parameters and hence on the mapmakers’ preconceptions - thus have large uncertainties that are generally underestimated and not communicated to public - sometimes either underpredict hazard in areas where large earthquakes occur - or overpredict hazard Without objective testing, maps won’t improve & seismology will keep having to explain away embarrassing failures

80. Challenge: Users Want Predictions Future Nobel Prize winner Kenneth Arrow served as a military weather forecaster. As he described, “my colleagues had the responsibility of preparing long-range weather forecasts, i.e., for the following month. The statisticians among us subjected these forecasts to verification and found they differed in no way from chance. The forecasters themselves were convinced and requested that the forecasts be discontinued. The reply read approximately: "The commanding general is well aware that the forecasts are no good. However, he needs them for planning purposes." Gardner, D., Future Babble: Why Expert Predictions Fail - and Why We Believe Them Anyway, 2010