1. Harnessing butterflies for seasonal and interannual Predictions
Stochastic Seasonal to Interannual Prediction System
Harnessing butterflies for seasonal and interannual Predictions
CAP, 13 June, 2016
S. Lovejoy, L. Del Rio Amador, McGill, Montreal

2. StocSIPS* with SLIMM**
Forecasts from months to decades:
The unsuspected Elephantine (“long range”) memory
StocSIPS* with SLIMM**
10% of the information needed for global seasonal temperature forecasts comes from fluctuations more than 300 years old…
But we can (almost) do it!
*StocSIPS= Stochastic Seasonal and Interannual Prediction System
**SLIMM= ScaLIng Macroweather Model

3. Statistical Mechanics
Low level (fundamental)
deterministic
stochastic
Vortices in strongly turbulent fluid
(M. Wiczek, numerical simulation, 2010)
Continuum mechanics
Thermodynamics
Higher level
Continuum limit
Fluctuations ≈ (turbulent flux) x (scale)H
Higher level
Laws of turbulence
High Reynolds number limit
Richardson, Kolmogorov, Corrsin, Obukhov, Bolgiano
Long time limit >10days
Space-time factorization
Macroweather laws

4. Temperature Change (oC)
Power laws
H=0.4
H=-0.4
H=0.5
H = -0.7
Temperature Change (oC)
Montreal
Weather
EPICA
(Antarctic)
±2K
±3K
Ice ages
2ox2o
(75oN)
5oC
Macroweather
macroclimate
2oC
10 days
1yr
2 hours
10,000 yrs
10 yrs
100 yrs
1 oC
1000 yrs
106
note the low freq. climate slope is about -0.8
100,000 yrs
0.5oC
climate

5. Trichotomy: Weather – macroweather - climate
H ≈ 0.4: Fluctuations Growing
H ≈ -0.4: Fluctuations Decreasing
= constant
Fluctuation
1 Century,
Vostok,
20-92kyr BP
Climate
( yrs to
50,000 yrs) 14
Temperature (oC) 12
20 days,
75oN,100oW,
Macroweather
(10 days to yrs) 2 4 6 8 10
1 hour,
Landers,
10 Feb.-12
March, 2005
0.067 s,
Rutherford Physics,
5:05 pm
Nov. 4, 2004
Weather
(up 10 days)
The series are nomlized and then displaced by 4 for clairty bottom is Landers (dertended yearly and daily), middle is 75N, long=100o 20 day averages, top is Vostok at 1 centruy res 20kysBP to 92kyrsBP
Sd’s: sdVostok= , sdlowfreq = , sdLanders =
orange is present with 20 day average
Blue is hourly, Feb 10 - March 12, 2005,
The bottom (new is from the 3.8m thermisotor at 15Hz; the s.d. was 0.35 K H=0.4 also
Ponk is 20kyrBP-92kyrBP -
100
200
300
400
500
600
700 t
Lovejoy, 2013, EOS

6. Beyond their deterministic limit: GCMs stochastic
“Brute force”
Weather systems (<10 days) generated by GCMs
= random weather noise (statistics)…
but not fully realistic
Averages: slow convergence to
Model climate
Scaling laws generate realistic (empirically based) statistics (noise)
Our climate
Potential advantages of direct stochastic macroweather (>1 month) forecasting:
More realistic weather “noise” (statistics: based on empirical data, not constrained by model).
Ability to use empirical data to force convergence to the real climate.

7. Statistical characteristics of Macroweather
Temporal domain
- Low intermittency Gaussian theory (except for extremes).
- Scale symmetry ≈ 1 month- >100 years (anthropocene ->30 years)
Fluctuations tend to cancel: H<0.
- Theoretical stochastic limits to forecast skill: theoretical “benchmark”
- Scale symmetry: huge memory
Spatial domain
- Scale symmetry up to ≈ 4000 km.
- Strong (multifractal) intermittency: climatic zones
Space-time
Statistical space-time Factorization.
Strong spatial correlations do not give useful information for forecasting: single grid or single station forecasts close to the theoretical maximum.

8. Statistical space-time Factorization
(Ex: factorization of second order statistics)
Spectral densities:
Weather, no factorization
Space-time Scale function
Typical form
Macroweather: factorization
Structure Functions:
Implies size – lifetime relation
Macroweather: factorization
No relation between size and lifetime

9. GISS E2R temperatures (historical run since 1850)
Factorization: GCM’s
GISS E2R temperatures (historical run since 1850)
Log10S(Δq)
Factorization:
Scaling:
Factorization implies parallel, identical curves Δt
0.5
1 months 2
Log10Δq 4 8 16
1.0
1.5
2.0 32 64
128 -
0.5
Fluctuations decrease as the time-averaged interval increases.
256
512 -
1024
1.0
1872
80o

10. Deterministic predictability limits for Weather forecasts
Sensitive dependence on initial conditions leads to limits (“butterfly effect”).
Fundamental limit = error doubling time.
The doubling time increases with the lifetime (hence size).
For planetary structures ≈ 10 days.
Weather forecast skill can be judged by how close the doubling time is to the 10 day limit.
This is the basic benchmark for deterministic weather forecasts

11. Stochastic predictability limits for Macroweather forecasts
Temporal scaling and statistical space-time factorization imply
stochastic predictability limits.
Skill = (forecast error variance)/(temperature variance)
Theoretical skill = F (lead time, H(x))
the exponent H(x) and hence skill varies with position x.
This is the basic benchmark for macroweather forecasts
-For 5ox5o resolution monthly and seasonal StocSIPS forecast skill =25%, 18%: theoretical limits = 29%, 21%, respectively.
-StocSIPS forecasts are on average 86% of the theoretical limit.
Comments:
-The skill has only a weak dependence on spatial resolution… hence we can avoid downscaling.
Can GCM’s improve on the theoretical stochastic limits?
Not obvious since the GCM’s appear to satisfy space-time factorization and temporal scaling

12. Hasselman (1976) type stochastic Macroweather processes
b=2
log10w
log10E(w)
(10 days)-1
b=0
macroweather
weather
Low frequency (b=0)
White noise (no predictability)
(Orenstein Uhlenbeck processes, Linear Inverse Modelling,
Auto Regressive processes)
The Hasselman (1976) type stochastic approach:
White noise
E(w)≈w-b

13. Scaling stochastic Macroweather processes
log10E(w)
red noise (potentially huge predictability)
White noise
Scaling, stochastic approach
(Ignoring intermittency)
-1/2<H<0 (fluctuation exponent)
b≈1+2H
b≈1.8
Empirical, turbulence
Empirical, macroweather
E(w)≈w-b
b=0
Hasselman type process
b=2
(10 days)-1
log10w

14. Gaussian white noise: H= -0.5
Skill
Skill as a function of forecast lead time
1.0 S k
Skill = 1- (Error variance)/(temperature variance)
0.8
0.6
58% (seasonal, hindcasts)
61% (monthly, hindcasts)
68% (theory analytical)
64% (theory numeric)
Global H =
Low frequency
LIM: H = -1/2
0.4
Dimensionless Forecast horizon/resolution l=t/t 1 2 4
0.2
Gaussian white noise: H= -0.5 64 -
0.5 -
0.4 -
0.3 -
0.2 -
0.1 H
H = 0: Pure “1/f” noise
Land
Ocean
Skill for horizons 1, 2, 4, 8, units away GOOD!!!!! Finf is good one

15. Fluctuations contributing to 90% of the theoretical maximum skill-
0.5
0.4
0.3
0.2
0.1 H 1 2 3 4
Log 10 l
mem
The unsuspected Elephantine (“long range”) memory
Fluctuations contributing to 90% of the theoretical maximum skill
Global
1200
Ex: with H = , fluctuations over the last 1200 seasons (300 yrs) are responsible for 90% of the seasonal variability
Ocean: PDO
600 50 15
the Log10 memory needed to give 90 % of the maximum skill at horizon = 1 forecast 9
White noise
Pure “1/f” noise

16. Preprocessing of the data:
Ref: (NCEP/NCAR) Example for the grid point (-72.5, 47.5), Montreal
Raw Data and Annual Cycle
Raw Data
Trend
Annual Cycle
Anomalies

17. 𝛽=1+2H Spectrum and Fluctuation Analysis
Spectra pre and after processing
annual peak
Montreal
𝛽=1+2H
Global spectra pre and after processing
annual peak
Global (36x72 pixels ensemble)
𝛽=0.46±0.02
Fluctuation Analysis
Global (36x72 pixels ensemble)
𝐻=0.27±0.01

18. “The ‘closest witnesses’ to the unobserved past have special weight”
Scaling LInear Macroweather model (SLIMM)
Prediction of fGn
Gaussian noise
Power law correlation. Vast memory that can be exploited.
Predictor for -0.5 < H < 0 based on past data.
Kernel for H = -0,1.
kernel
Weight
of present
predictor
data
Weight of the distant past
“The ‘closest witnesses’ to the unobserved past have special weight”

19. Theoretical and numerical Skills. Monthly resolution.
Period Sep, Dec, Reference: ERA-Interim Reanalysis
Theory MSSS, lead time = 1 month
<r> = 0.95
1-r
Numerical MSSS, lead time = 1 month

20. StocSIPS relative advantage increases with lead time
A = Percentage of the globe where
MSSSStocSIPS.MSSSCanSIPS
Diff=MSSSStocSIPS-MSSSCanSIPS
lead time = 1 month, A=26%
lead time = 2 months, A=55%
lead time = 6 months, A=65%
lead time = 9 months, A=69%

21. Skills StocSIPS and CanSIPS: Comparison
Annual resolution, lead time = 2 years for StocSIPS. Monthly resolution, lead time = 6 months for CanSIPS
MSSS StocSIPS, annual, lead time = 2 years
MSSSStocSIPS (2 years) - MSSSCanSIPS(6 months)
𝐴 =69%
MSSS CanSIPS, monthly, lead time = 6 months
<Diff>=0.13
sdiff=0.22

22. Relative Skill of StocSIPS increases with lead time
MSSSStocSIPS>MSSSCanSIPS for more than one month
MSSSglobal = 1-<MSE>/Varanom
Mean Square Error
𝑀𝑆𝐸 -
Varanom = Variance of the anomalies

23. Global Producing Centers: Actuals
GPC- Beijing, JFM
GPC-Exeter
Anthropogenic
Natural variability
Actuals
For actuals
(negative skill)
GPC-Melbourne
GPC-Toulouse
GPC-Montreal
GPC-Washington
𝑴𝑺𝑬=𝟓𝟎𝟎 % 𝐨𝐟 𝑽𝒂𝒓 𝒂𝒏𝒐𝒎

24. Skills StocSIPS and CanSIPS: Actuals
MSSS StocSIPS, monthly, lead time = 1 month
MSSS StocSIPS, annual, lead time = 1 year
MSSS StocSIPS, seasonal, lead time = 1 season
MSSS CanSIPS, monthly, lead time = 1 month

25. Hindcasts Skill: Actuals
Diff=MSSSStocSIPS--MSSSCanSIPS
𝐴 =93%
MSSSglobal = 1-<MSE>/Varanom
<Diff>=3.7
Anthropogenic
Natural variability
Actuals
For actuals
MSE>Var (anom)
(negative skill)

26. Probabilistic Forecast
Forecast probability
Climatological probability
If all probabilities < 40% then grey

27. Probabilistic Forecast: verification and validation
Global average PC = 48,5% (Same as CanSIPS for Canada)
A = 95% of the globe with PC > 33%

28. Visit our site

29. Conclusions Theoretical basis of StocSIPS StocSIPS performance
-Long term memory (scaling, one parameter, H)
-Space-time factorization (space-time prediction decoupling)
-Stochastic predictability limits
StocSIPS performance
-Anomalies: StocSIPS has higher skill than CanSIPS for two months and longer.
-Actuals: Higher skill at all lead times due to direct forecasting of climatology.
-StocSIPS relative advantage: increases with lead time and is higher over land than oceans.
StocSIPS’ advantages include
-No data assimilation
-No ad hoc post processing
-No need for downscaling
-Speed: (factor )
Future developments
-Prediction of other fields (precipitation, wind, solar insolation, degree-days, forest fire indices, drought indices).
-Find co-predictors such as El Niño indices.
-Prediction of Extremes