1. Validity of climate change forecasting for public policy decision making
Kesten C. Green
Business and Economic Forecasting Unit
Monash University
J. Scott Armstrong
The Wharton School
University of Pennsylvania
9 March, 2009 1

2. Characteristics of temperature series
Temperature varies over time, but
No persistent trend
Apparent short and long trends
Of varied length
That reverse without warning 2

3. Antarctic temperature changes3

4. More temperature history
Second, a shorter global approximation from thermometer data… 4

5. Hadley annual temperature 1850-2008

6. Conditions favor conservatism
Many opinions by experts, but no evidence that the climate is different now 6

7. No-change benchmark model
Tempyear+1,2,…100 = Tempyear
Consistent with the viewpoint of
a weather-dominated climate system 7

8. Forecasting methodology Structuring the forecasting problem
Auditing the forecasting process
Selecting a forecasting method
Validation tests 8

9. Test of the benchmark
Used HadCRUt3 “best estimate” of global mean temperatures from 1850 to 2007
Each year’s temperature a forecast for up to 100 subsequent years
157 one-year-ahead forecasts…
58 hundred-year-ahead forecasts
10,750 forecasts across all horizons
Absolute errors calculated vs HadCRUt3 9

10. 10

11. 11

12. Validity of IPCC projection
1992 IPCC Report projected 0.03oC/year linear
Test vs benchmark for
1992 to 2008
1851 to 1975
1992 to 2008; ex ante
1851 to 1975; analogous exponential CO2 growth 12

13. IPCC performance 1992-2008 (short-range ex ante)
CumRAE IPCC/Benchmark*
Hadley UAH n
From
Rolling
* CumRAE < 1 means forecast errors smaller than benchmark errors 13

14. Mean errors 1992-2008 Using UAH data and rolling forecasts,
Averaging the mean absolute errors for all 17 horizons…
Benchmark oC
IPCC projection oC
Difference oC 14

15. IPCC performance 1851-1975 (long range; ex post IPCC advantage)
CumRAE of IPCC/Benchmark
Hadley n
From
Rolling (1-100 years) ,550
1-10 years ,205
41-50 years
years
* CumRAE < 1 means forecast errors smaller than benchmark errors 15

16. Armstrong-Gore bet expectations
Based on the Hadley data for 1850 through 2008*…
Armstrong has a probability of wining bets against the 0.03oC/year linear trend of
0.54 for forecasts one-year-ahead; n=158
0.68 for ten-year-ahead forecasts; n=149
* which, as we know, was a warming period 16

17. Antarctic temperatures revisited17

18. Forecasting methodology Structuring the forecasting problem
Auditing the forecasting process
Selecting a forecasting method
Validation tests 18

19. Alternatives to the benchmark
Predictions from outcomes of analogies
Causal model with atmospheric CO2
Policy implications?
Rival causal variables? 19

20. Why Structured Analogies?
Analogies commonly used to “sell” forecasts. They do not aid accuracy when used in this way.
Structured analogies produce better forecasts by overcoming biases. They help learning from history and thereby improve accuracy.
(Green & Armstrong 2007) 20

21. Structured analogies procedures
Ask heterogeneous group* of experts to individually describe as many analogies as they can for the current AGW situation.
Experts then rate analogies for similarity to the AGW situation.
Mechanical summary: Forecasts based on the set of “most similar” analogies.
* Recruit warmers, skeptics, and others. 21

22. List all analogies that have the following features
Popular movements that were endorsed by scientists and politicians. Include all (i.e., those that proved to be justified by events as well as those that weren’t)
Expert, lobby group, and media interest high
Supported by some scientists
Accompanied by calls for government action
Outcomes now known and agreed upon
Please use the sign up sheets if you are willing to participate. Thank you! 22

23. Causal model testing procedures
Data: Global mean temperature: Hadley
CO2: Total global atmospheric concentration, NASA
Models: First differences; levels
Estimated initially using data
Forecasts: Annual
Rolling forecasts for up to 100 years
Updated estimate of relationship each roll
Conditional on knowing CO2
Accuracy: CumRAE, or cumulative relative absolute error;
Sum forecast errors, ignoring signs, and
divide by the equivalent sum of benchmark model
errors
[Please write suggestions on the page we circulated]

24. Causal models
Forecast temperature using CO2 concentration versus alternatives explanations
Over the period 1850 to 2008
Forecasts conditional on knowing future atmospheric CO2
CumRAE for annual forecasts with year horizon (CumRAE is the absolute error vs. the benchmark error)
Model: Please rank in terms of forecast accuracy
___ 1. Policy: CO2 causes warming; (t; t+1)
___ 2. Effect not cause: Warming causes CO2; (t; t-1) 24

25. Causal model Direction 1. CO2 policy – 2. CO2 effect not cause +
Tentative results: conditional forecasts Signs on coefficients for 1diff lagged models
Causal model Direction
1. CO2 policy –
2. CO2 effect not cause

26. Causal model CumRAE 1. CO2 policy 0.83 2. CO2 effect not cause n/a
Tentative results: conditional forecasts (based on 5,950+ forecasts from levels models)
Causal model CumRAE
1. CO2 policy
2. CO2 effect not cause n/a

27. CO2 policy implications? Or not?
Assume that if
Manmade CO2 emissions contribute a net 0.3ppm/year to the atmosphere
Policies could reduce that by 20% to 0.24ppm/year
…and sustain that reduced rate for 100 years
Then, our tentative first round results suggest:
First difference model suggests that net 6ppm not contributed to atmosphere would result in a global mean temperature 0.02oC higher in 100 years.
Levels model suggests the mean temperature would be 0.05oC lower in 100 years.
Alternatively, stopping human emissions altogether would increase mean temperature by 0.40oC, or decrease it by 0.24oC, over 100 years

28. Might CO2 be spurious?
Some scientists believe the hypothesized strong CO2–temperature relationship to be spurious.
Moreover, relatively smaller forecast errors produced by a model of temperature changes causing subsequent CO2 changes. 28

29. Summary
Policy decisions require scientific long term forecasts of temperature and effects of policies
Such forecasts do not exist
Climate data and knowledge are uncertain, and climate is complex
The situation calls for simple methods and conservative forecasts
The no-change benchmark performs well
IPCC projections do not compare well
Do causal policy models with CO2 lack credibility?
What do analogies imply for the future of AGW?
Scientific forecasting suggests appropriate policy decision is “don’t just do something, stand there!”

30. Handouts for Armstrong & Green talks
References for papers cited in our talks /
WWF solicitation based on False advertising?
Boxer vs.Armstrong Charity Wager
Gore vs. Armstrong Prediction Market.
Suggestions on what to do?
• Analogies study
• WWF action steps
• CO2 policy forecasting
“Climate Change Forecasts are Useless for Policy Making”

31. References for the Armstrong and Green talks
[Papers available at unless otherwise indicated]
Armstrong, J. S., Green, K. C., & Soon, W. (2008). Polar Bear Population Forecasts: A Public-Policy Forecasting Audit. Interfaces, 38, 5, 382–405. (includes commentary & response).
Graefe, A., Armstrong, J. S., & Green, K. C. (2009). Using Prediction Markets to Solve Complex Problems: An Application to the ‘Climate Bet’. Working paper available at
Green, K.C., & Armstrong, J. S. (2007). Global Warming: Forecasts by Scientists versus Scientific Forecasts. Energy and Environment, 18, No. 7+8,
Green, K.C., & Armstrong, J. S. (2007). Structured Analogies in Forecasting. International Journal of Forecasting, 23,
Green, K.C., Armstrong, J. S., & Soon, W. (2009). Validity of Climate Change Forecasting for Public Policy Decision Making. International Journal of Forecasting, Forthcoming.
Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press, Princeton, NJ.
Gore-Armstrong prediction market at
Details on Gore-Armstrong bet at