2. Shallow versus deep uncertainties Why many predictions / forecasts fail

Society is playing a high-stakes game of chance against nature We want to - assess the hazard - how often dangerous events happen - mitigate or reduce the risk - the resulting losses. Often nature surprises us, when an earthquake, hurricane, or flood is bigger or has greater effects than expected from hazard assessments. In other cases, nature outsmarts us, doing great damage despite expensive mitigation measures, or causing us to waste resources on what proves a minor hazard.

Hazard assessment failed 2010 map predicts probability of strong shaking in next 30 years But: 2011 M 9.1 Tohoku, 1995 Kobe M 7.3 & others in areas mapped as low hazard In contrast: map assumed high hazard in Tokai “gap” Geller 2011

Hazard model divided trench into segments Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2 (Headquarters for Earthquake Research Promotion)

Giant earthquake broke many segments 2011 Tohoku Earthquake 450 km long fault, M 9.1 (Aftershock map from USGS) J. Mori Expected Earthquake Sources 50 to 150 km segments M7.5 to 8.2 (Headquarters for Earthquake Research Promotion)

Tsunami runup approximately twice fault slip (Plafker, Okal & Synolakis 2004) M9 generates much larger tsunami Planning assumed maximum magnitude 8 Seawalls 5-10 m high CNN NYTStein & Okal, 2011

NY Times 3/31/2011 Mitigation failed Expensive seawalls - longer than Great Wall of China -proved ineffective Tsunami overtopped 10m high sea walls, causing more than 15,000 deaths and $210 billion damage.

What’s going wrong? Shallow uncertainty - we don’t know what will happen, but know the odds (probability density function). The past is a good predictor of the future. We can make math models that work well. Deep uncertainty - we don’t know the odds. The past is a poor predictor of the future. We can make math models, but they generally won’t work well.

Shallow uncertainty is like estimating the chance that a batter will get a hit. His batting average is a good predictor. Deep uncertainty is like trying to predict the winner of the World Series five years from now. Teams' past performance give only limited insight into the future.

Due to deep uncertainty Predicted natural or other disaster probabilities are hard to estimate and thus often very inaccurate The world is more complicated than we think or admit Prob(sinking) = 0

1986 – Loss of shuttle Challenger NASA claimed probability of loss = 1/100,000 Richard Feynman argued for 1/100 – 1000 times higher. As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... "

1986 – Loss of shuttle Challenger NASA claimed probability of loss = 1/100,000 Richard Feynman argued for 1/100 – 1000 times higher. As he pointed out in his report dissenting from the government investigation commission, because this rate implies that "one could put up a shuttle every day for 300 years expecting to lose only one, we could properly ask what is the cause of management's fantastic faith in the machinery... " In 2003, shuttle Columbia was lost on the 107th shuttle mission 2 lost in 107 missions ≈ 1/50

Activity 2.1: Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000?

Activity 2.1: Given that the shuttle was a new spacecraft, before flights started, how could you assess NASA’s estimated probability of loss = 1/100,000? One way: of 11 Apollo missions, one (Apollo 13) suffered near disaster. In addition, Apollo 1 was lost in a launch pad fire.

Boeing 787 Dreamliner batteries Boeing “concluded that they were likely to emit smoke less than once in every 10 million flight hours. Once the planes were placed in service, the batteries overheated and emitted smoke twice, and caused one fire, after about 50,000 hours of commercial flights.” (NYT, 2/7/13)

From 1975 to 2007, U.S. housing prices grew steadily. Neither Washington nor Wall Street recognized that this could not go on forever, or worried that trillions of dollars of risky mortgages were embedded throughout the financial system

As housing prices and subprime lenders collapsed, Wall Street & government weren’t concerned based on computer models

The models were wrong

NY Times 3/21/11

Hazard maps are hard to get right: successfully predicting future shaking depends on accuracy of four assumptions over years Where will large earthquakes occur? When will they occur? How large will they be? How strong will their shaking be? Uncertainty & map failure result because these are often hard to assess, given that the earthquakes are much more variable in space and time than the short earthquake history shows

2008 Wenchuan earthquake (Mw 7.9) was not expected: map showed low hazard based on lack of recent earthquakes Didn’t use GPS data showing 1-2 mm/yr (~Wasatch) Earthquakes prior to the 2008 Wenchuan event Aftershocks of the Wenchuan event delineating the rupture zone

GSHAP 1999 NUVEL-1 Argus, Gordon, DeMets & Stein, 1989 Swafford & Stein, 2007 Slow plate boundary Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

Swafford & Stein, 2007 GSHAP 1999 NUVEL-1 Argus, Gordon, DeMets & Stein, 1989 M 6.4 M 6.3 Slow plate boundary Africa-Eurasia convergence rate varies smoothly (5 mm/yr)

Familiar pattern 2001 hazard map M7 earthquake shaking much greater than predicted for next 500 years

Italian hazard maps, which predicted the expected shaking in the next 500 years, forecast some earthquake locations well and others poorly, and so required updating within a decade.

A posteriori changes to a model are "Texas sharpshooting:” shoot at the barn and then draw circles around the bullet holes.

Overfitting: a subtle trap We can fit very complicated models to data like earthquake histories, but we are partly fitting noise Flipping a coin gives lots of complicated patterns H H T T H T T T H T H H T H We could fit a model to those data, but it would do no better than 50% at predicting the next flip

Overfitting: a subtle trap We can fit very complicated models to data like earthquake histories, but we are partly fitting noise In such cases, a more complicated model can give worse predictions

Activity 2.2: Time between large earthquakes from paleoseismic record on southern San Andreas What’s the mean time between large earthquakes here? When would you expect the next one? Is one due soon, overdue, or… DD 9.8

Time dependent predicts lower until ~2/3 mean recurrence Results depend on both model choice & assumed mean recurrence Hebden & Stein, 2008 We don’t know whether to assume that probability of a major earthquake is - constant with time (time-independent) or - small after a large earthquake and then increases (time-dependent ).

Activity 2.3 Deep uncertainty in earthquake recurrence Imagine an urn containing e balls labeled "E" for earthquake, and n balls labeled "N" for no earthquake. We can draw balls in two ways.

Activity 2.3 Deep uncertainty in earthquake recurrence Option 1: after drawing a ball, we replace it. In successive draws, the probability of an event is constant or time- independent. Because one event happening does not change the probability of another happening, an event is never overdue.

Activity 2.3 Deep uncertainty in earthquake recurrence Option 2: We can add a number a of E-balls after a draw when an event does not occur, and remove r E- balls when an event occurs. This makes the probability of an event increase with time until one happens, after which it decreases and then grows again. Events are not independent, because one happening changes the probability of another.

Activity 2.3 Deep uncertainty in earthquake recurrence Problem: Given a sequence of results, it’s hard or impossible to tell how the urn was sampled. Thus it’s hard to assess the probability of an “earthquake” in the next draw.

Italian flag graphic - one way to illustrate uncertainty we can’t quantify well 50% 30% 20%