1. Overview of the RHtests_dlyPrcp software package for homogenization of daily precipitation Xiaolan L. Wang and Y. Feng EMS 2015, Sofia, Bulgaria, 7-11 September 2015 Climate Research Division, Science & Technology Branch, Environment Canada

2. FindU FindUDStepSize FindU is for detecting unknown mean shifts - in constant trend series with Gaussian IID or AR(1) errors (red noise); - can be used without reference series FindUD and StepSize are for testing the significance of documented shifts without a ref. series Its sister package for Gaussian data – RHtestsV5 (Poster P75) - consists of six major functions: FindU.wRef FindUD.wRefStepSize.wRef FindU.wRef is for detecting unknown mean shifts - in zero-trend series with Gaussian IID or AR(1) errors (red noise); - for use with reference series FindUD.wRef and StepSize.wRef are for testing the significance of documented shifts with a ref. series These and other existing methods cannot be directly used to homogenize daily precipitation data, because daily precipitation data are - non-continuous (precipitation does not occur every day at a fixed location) - non-negative (no negative values) - non-normally distributed, and - highly variable spatially, which makes it hardly possible to find/use a reference series The amounts and occurrence frequencies must be homogenized, separately. It is wrong to adjust zero daily precip. amounts for days of no precipitation occurrence!

3. 3 RHtests_dlyPrcp The RHtests_dlyPrcp software package is specifically designed for, and is the only software that is suitable for homogenization of daily precipitation data: - It finds the best Box-Cox transformation for the non-zero precipitation amounts in question, and applies the transformation to bring the data close to a normal distribution. - It applies the PMFred algorithm to the transformed data to test for unknown shifts (i.e., transPMFred), and the F test, to test the significance of documented shifts - It provides Quantile-Matching (QM) adjustments and IBC adjustments for the determined shifts in the daily precipitation amounts Details in Wang et al. 2010 ( J. Appl. Meteor. Climatol., 49, 2416-2436) “mean” adjustments based on Inverse Box-Cox transformation

4. The Box-Cox transformation improves the detection power and lowers the FAR (Each case: generated 1000 series of N=1000 from a Log-Normal or Gamma distribution; see Wang et al. 2010) OriginalTrans’dOriginalTrans’d TPR30.0780.0510.0670.044 PMF0.0700.0550.0710.047 False Alarm Rates (FARs): (nominal significance: 0.05) Hit rates: Log-Normal data transPMFred transTPR3 transPMFred transTPR3 transPMFred transTPR3 PMFred TPR3 PMF TPR3 PMFred TPR3 TPR3 – a maximum F test used on a common trend two-phase regression model for larger shifts for small shifts Power increases for Gamma distributed data  Log-Normal Gamma Power increases for Log-Normal distributed data 

5. The software RHtests_dlyPrcp consists of three functions; all of them available in GUI mode: Parameters used currently must be changed to the code used for missing values in your data! You can change these default values to the values you want to use. Namely, you can choose (1) the significance level to conduct the test, (2) the segment to which to adjust the series, (3) the number of categories/points you want to use to estimate the probability distribution, (4) to use all or part of the data in a segment to estimate the QM adjustments You can also choose to test only precip. values that are greater than a chosen threshold, say 0.5 mm FindU Click FindU button to choose the precip. series to be tested. Then, click Ok to run the test. This will find significant Type-1 (unknown) changepoints, i.e., those that are significant even without metadata support FindUD Click FindUD button to find potential Type-0 changepoints, namely, those that are significant only if they are supported by metadata. Skip this step if you don’t have metadata or only want to focus on Type-1 shifts to adjust the data to the latest seg. (better to adjust to the highest seg.) StepSize Click StepSize button to re-estimate the size and significance of shifts after you make any change in the list of changepoints, for example, add a documented shift, or change the date to a nearby documented date of change, or decide not to adjust a statistically detected shift…

6. Different stages of further adjustments  four different daily precip. data series: 1. Not incl. trace amounts, no adjustment for joining (noT_naJ) 2. Not incl. trace amounts, adjusted for joining (noT_aJ) 3. Incl. trace amounts, no adjustment for joining (wT_naJ) 4. Incl. trace amounts, adjusted for joining (wT_aJ) by Vincent & Mekis (2009), using one rainfall ratio & one snowfall ratio for all data in a segment Daily precipitation recorded at The Pas (Manitoba, Canada) for Jun 1 st, 1910 to Dec 31 st, 2007 - snowfall  water equivalent; rainfall adjusted for wetting loss and gauge undercatch (Mekis & Hogg 1999; and updates by E. Mekis) - joining of two stns at the end of 1945 (5052864 for up to 31 Dec. 1945, 5052880 1 Jan 1946 to 31 Dec. 2007) Next, I’ll show you: same three changepoints detected Same two changepoints detected Examples of application All four series have a very significant changepoint near the time of joining!  The ratio-based adjustments failed to homogenize the data series!

7. Results for the two series not including trace amounts (noT series): Type Date Documented date of change(s) 1 4 Jul 1938 9 Oct 1937 to 8 Aug 1938: changes in gauge type, rim height, observing frequency; poor gauge condition reported on 9 Oct 1937 1 24 Oct 1946 31 Dec 1945: joining of two nearby stations (5052864 + 5052880) 1 4 Oct 1976 16 Oct 1975 to 18 Oct 1977: gauge type change (standard at 12” rim height to Type B at 16” rim height) Reflect changes in the min. measurable amount (precision, unit) 1976-77 1945-46 joining 1937-38 The transPMFred detected the same 3 changepoints. All of them are supported by metadata: noT_naJ -0.76 mm 1. noT_naJ (closest to original measurements): 2. noT_aJ (aJ changed the mean shift size from -0.76 mm to -0.73 mm) The ratio-based adjustments for station joining failed to homogenize the series, because … joining

8. The discontinuities are mainly in the measurements of small precipitation (P ≤ 3 mm), especially in the frequency of measured small precipitation: Series of daily P > 3 mm is homogeneous! noT_naJ > 3mm No P < 0.3 mm before 1977 (precision changed) noT_naJ Much fewer 0.3 ~ 0.5 mm before 1946 -joining point 0.21 mm from SWE Much fewer < 3.0 mm before 1938 Good news for studying extreme precipitation Any ratio-based adjustments for joining are not good in this case, because larger P are adjusted more than smaller P when they should not be adjusted at all! noT_aJ The above freq. discontinuities largely remain:

9. Type Date Documented date of change(s) 1 29 Jun 1931 early 1930s: MSC gauge intro’d; (new) 9 Oct 1931: noticed the need to relocate the gauge in order to collect a correct rainfall 1 19 Nov 1945 31 Dec 1945: joining of two nearby stations (5052864 + 5052880) wT_naJ Inclusion of trace amounts makes these shifts disappeared! transPMFred detected the same 2 changepoints, consistent with metadata: 3. wT_naJ 4. wT_aJ Results for the two series including trace amounts 1976-77 1937-38 joining

10. An example of frequency discontinuity The frequency of reported trace occurrence at The Pas is not homogeneous! Adding a trace amount for T-flagged days is not good enough Need to address the issue of frequency discontinuity! Otherwise, adjustments could make the data deviate more from the truth! noT_naJ 1945-46 1955-56 No trend

11. How to address the issue of frequency discontinuity? FindUFindU.wRef - can use FindU or FindU.wRef in the RHtestsV5 package (poster P75) to test the annual or monthly frequency series of some event, e.g., trace occurrence: noT_naJ 1945-46 1955-56 FindU, or FindU.wRef FindU, or FindU.wRef with the long-term mean frequency (a constant value) as the reference series

12. Concluding remarks should be chosen to reflect changes in measurement precision/unit - Homogenization of daily precipitation data is very challenging Recommend: test series of dailyP > Pthr with different Pthr values (e.g., 0.0, 0.3 mm, 0.5 mm, 1.0 mm) Shall aim to get better insight into the cause (metadata) and characteristics of discontinuity (e.g., freq.) before any attempt to adjust daily precipitation data – a non-continuous variable! also check the frequency series of small P, because small P are harder to measure with accuracy than larger P (larger %error) – discontinuities often exist in freq series of small P (e.g. P < 0.5 mm) In the presence of frequency discontinuity, any adjustment derived from the measured daily P is not good. (e.g., ratio-based, IBC, Quantile-Matching) One must address the issue of freq. discontinuity first!

13. Thank you very much for your attention! Questions/comments? The package is available at http://etccdi.pacificclimate.org/software.shtml http://etccdi.pacificclimate.org/software.shtml