Break Position Errors in Climate Records Ralf Lindau & Victor Venema University of Bonn Germany

12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Internal and External Variance Consider the differences of one station compared to a neighbor reference. Breaks are defined by abrupt changes in the station-reference time series. Internal variance within the subperiods External variance between the means of different subperiods Break criterion: Maximum external variance

12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Decomposition of Variance n total number of years N subperiods n i years within a subperiod The sum of external and internal variance is constant.

Position errors Two segments of lengths n 1 and n 2 with means x 1 and x 2. A subsegment of length m with mean x 0 is erroneously exchanged from segment 2 to segment 1. x 1 is strongly reduced, x 2 differs slightly. x 1 and x 2 converge. This reduces the external variance, and the wrong segmentation is rejected. 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Change of external variance 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 The change of external variance  v is only a function of the means and lengths of the two segments and the exchanged subsegment.

Express x 0 by x 2 plus scatter 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 The mean of the exchanged subsegment x 0 is equal to x 2, the segment mean where it stem from, plus a random scatter variable   depends on the internal variance  2 and the length m, because it is a mean over m random numbers. with

Quadratic function for  v 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Replace x 0 by  and normalize by the square of the jump height d. The change of the normalized external variance v*, which is the decision criterion for break detection, is a quadratic function of a random variable  which depends on the signal to noise ratio and the length of the exchanged segment.

Zero points If the parabola becomes positive, the shift of the break position by m items leads to increased external variance so that this solution is preferred by mistake. Zero points at: 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Simulated data 10,000 random time series of length 100. Internal  = 1 Jump height = 2 Data confirm the existence of different parabolae for different m. But data coverage only for scatter near zero, never reaching the negative solution. 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 m=1 m=2 m=3  (n  v) / 4 } SNR = 1

The negative solution Typical situation: SNR extreme low. A drastically disturbed measurement near the break. Its exchange leads to x 1 ’ x 1. The two means diverge so that the external variance grows. 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 X1 X1’X1 X1’ X2’ X2X2’ X2

The positive solution A subsegment adjacent to the true break is randomly lifted by more than half of the jump height. Including it to the neighboring segment will reduce the internal variance. An erroneous break position is concluded. Criterion: Maximum hatched area 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Brownian motion with drift 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Drift = - SNR d 

Theoretical retrace 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Distribution of the time of the maximum of a Brownian motion with drift Strictly valid only for continuous processes. 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Buffet, 2003, J Appl Math Stoch Anal _ _ _ _ _ Buffet, Numerical simulation of a discrete Brownian motion with drift Complete break search simulation SNR = 0.5 SNR = 1SNR = 2

Two more problems 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Buffet, 2003 Hit rate is not accurately reproduced Break errors are a two-sided symmetric process. Both, too early and too late breaks are possible.

Hit rate The hit rate h can be estimated for all drifts d by: 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 true estimated

Two-sided processes 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Practical application The hit rate drops from from 95% for SNR = 2 to 29% for SNR = 0.5 SNR > 1  becoming quickly very exact. SNR < 1  becoming quickly very inexact. 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 true estimated SNR = 1 SNR = 2 SNR = 0.5

12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013 Conclusions Break position errors can be described by the distribution of the time of maximum of a Brownian motion with drift. The drift parameter is equal to the signal to noise ratio, as given by the half jump height between and the internal standard deviation within homogeneous subperiods.

Hit rate simulation The hit rate is the probability that the initial value is never exceeded. For realistic drift sizes the value converges after a few steps. 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Preliminary maximum 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013

Hit rate estimate 12 th International Meeting on Statistical Climatology, Jeju, Korea – 24. June 2013