Anthony Arguez NOAA National Climatic Data Center Phone: (828) OPTIMAL NORMALS On Improving NOAA’S Climate Normals: An Introduction to ‘Optimal Normals’ of Temperature Tuesday, June 2, 2009

Optimal Normals: Brief Overview A suite of Experimental Products that supplement the Traditional 30-Year Normals A suite of Experimental Products that supplement the Traditional 30-Year Normals Monthly Temperature (Max/Min/Mean) for now Monthly Temperature (Max/Min/Mean) for now Version 1.0 (computed through 2008): Version 1.0 (computed through 2008): –Annual Updates –OCN –Hinge Fit Later Versions Later Versions –More advanced techniques –More variables –Improved data source (discussed by Dr. Menne) OPTIMAL NORMALS Tuesday, June 2, 2009

This webcast is co-hosted by the AMS Energy Committee and NOAA’s National Climatic Data Center –Jon Davis, Chesapeake Energy: Chair of the AMS Energy Committee –Anthony Arguez, NCDC: project lead on Optimal Normals, NCDC’s User Engagement Lead for Energy, Member of the AMS Energy Committee. –Matthew Menne, NCDC: climate scientist involved in the creation of the dataset used for Optimal Normals OPTIMAL NORMALS Tuesday, June 2, 2009

Call for Papers –First Conference on Weather, Climate, and the New Energy Economy –American Meteorological Society Annual Meeting –17–21 January 2010, Atlanta, Georgia –More information? Contact Jon Davis or visit OPTIMAL NORMALS Tuesday, June 2, 2009 You may be interested in this….

History of this Project –Letters, informal discussion, anecdotal evidence: Is the 30-year Normal the best we can do? –May 2007 Teleconference: Listening –September 2007 Webcast: Proposal –June 2009 Webcast (now): Produce –Future:  Feedback, Perpetual Engagement  Release of Optimal Normals to all users OPTIMAL NORMALS Tuesday, June 2, 2009

OPTIMAL NORMALSWednesday, January 14, 2009 Traditional Climate Normals Issues in a Changing Climate Two main issues: –Is the 30-year average representative of the current state of the climate?  Consider this: Normals are only updated every 10 years! –What if there is a prominent trend? Are they obsolete? Is an average the best method? Climate Normals are calculated retrospectively, but are used prospectively for planning

Why Optimal Normals? Explicit Acknowledgment: No method is always perfect for all applications Explicit Acknowledgment: No method is always perfect for all applications Provide alternatives to the Traditional 30-Year Normals Provide alternatives to the Traditional 30-Year Normals –Experimental Products –Supplement, Not Replace, 30-Year Normals Evaluate these alternatives (later) Evaluate these alternatives (later) NOAA Leadership on this issue NOAA Leadership on this issue Livezey et al. (2007) Recommendations Livezey et al. (2007) Recommendations –Version 1.0 essentially follows these recommendations OPTIMAL NORMALSWednesday, January 14, 2009

OPTIMAL NORMALSWednesday, January 14, 2009 Gray: Not Significant

OPTIMAL NORMALSWednesday, January 14, 2009 Gray: Not Significant

OPTIMAL NORMALS Annual Updates A moving, or rolling, 30-year average A moving, or rolling, 30-year average Why? Official Normals are only computed once per decade. Why? Official Normals are only computed once per decade. e.g., instead of e.g., instead of Computed once per year (every January) → Update Computed once per year (every January) → Update Tuesday, June 2, 2009 Still an average

OPTIMAL NORMALS OCN Tool developed by NOAA’s Climate Prediction Center in the 1990s – they called it ‘Optimal Climate Normals’ Tool developed by NOAA’s Climate Prediction Center in the 1990s – they called it ‘Optimal Climate Normals’ Determine the ‘optimal’ averaging period: N Years Determine the ‘optimal’ averaging period: N Years the maximum correlation between the forecast anomaly and observed anomaly during the verification period the maximum correlation between the forecast anomaly and observed anomaly during the verification period Initially utilized in a sub-optimal fashion: fixed averaging periods, 10-year average for monthly temperature, 15-year average for monthly precipitation Initially utilized in a sub-optimal fashion: fixed averaging periods, 10-year average for monthly temperature, 15-year average for monthly precipitation ‘Optimal’ averaging period (N) can be computed per station, per variable, per monthly time series based on the residual lag-1 autocorrelation (g) and the linear trend (β). ‘Optimal’ averaging period (N) can be computed per station, per variable, per monthly time series based on the residual lag-1 autocorrelation (g) and the linear trend (β).  Livezey et al (JAMC) Tuesday, June 2, 2009 Still an average

OPTIMAL NORMALS Hinge Fit Piecewise continuous with no change from and linear change thereafter. Piecewise continuous with no change from and linear change thereafter. Modeled after underlying global warming signal. Modeled after underlying global warming signal. Reduces sampling error greatly from linear fit to last three decades. Reduces sampling error greatly from linear fit to last three decades. Outperforms the linear fit, and OCN except for small trends. Outperforms the linear fit, and OCN except for small trends. top provided by Bob Livezey  The line represents a time-dependent normal – there is no average involved Not an average Tuesday, June 2, 2009

OPTIMAL NORMALS Data and Methods: Overview We use data from We use data from stations in total 9168 stations in total Flags indicate at least 20% of values were interpolated Flags indicate at least 20% of values were interpolated All methods are applied to annually-sampled monthly time series All methods are applied to annually-sampled monthly time series –e.g. a January time series Tuesday, June 2, 2009

Data Processing Steps 1.Start with daily data sources (DSI-3200; DSI-3206; DSI-3210) Apply quality assurance (QA) checks; compute monthly values when no more than 9 daily values are missing 2.Merge these monthly values together to form one “superset” of monthly data Apply additional QA checks to the monthly values 3.Apply algorithm to adjust for bias associated with changes to the time of observation (all monthly values set to conform to a midnight to midnight observation hour) 4.Apply adjustments to account for changes in instrumentation, station moves, etc. 5.Create estimates for missing and/or flagged values using values from surrounding stations (FILNET) Each of these steps is described on the U.S. HCN Version 2 web site and in a forthcoming article Menne, Williams and Vose (2009) Bulletin of the American Meteorological Society (for early online release see /preprint/2009/pdf/ _2008BAMS pdf )

U.S. HCN Processing Steps are applied to the full Cooperative Observer Network to produce Normals dataset

The Time of Observation Bias

1950s 1960s 1980s s 1990s Hour of observation histograms at bottom of each U.S. decadal map (Figure courtesy of Xioamoa Lin, University of Nebraska)

Impact of Time of Observation Adjustments Average year by year difference over the conterminous United States between the Time of Observation Bias (TOB)-adjusted data and the unadjusted (raw) data.

Homogenization

Chula Vista annual maximum temperature departure from long-term average minus the average from 10 nearby stations. The Chula Vista station moved on January 1, 1982 (from 32°36'N, 117°06'W, Elev 9 feet to 32°36'N, 117°06'W, Elev 56 feet). °F Year Unadjusted Adjusted

(a) Mean annual unadjusted and fully adjusted minimum temperatures at Reno, Nevada. Error bars indicating the magnitude of uncertainty (±1 standard error) were calculated via 100 Monte Carlo simulations that sampled within the range of the pairwise estimates for the magnitude of each inhomogeneity; (b) difference between minimum temperatures at Reno and the mean from its 10 nearest neighbors

Maximum Temperature Trends – Raw (unadjusted) (1950 to 2007)

Maximum Temperature Trends – TOB Adjusted (1950 to 2007)

Maximum Temperature Trends – Fully Adjusted (1950 to 2007)

Future –Our plan is to vertically integrate future updates of the Optimal Normals monthly data with the Global Historical Climatology Network – Daily dataset ( –This will result in changes to the periods of record at some stations; however, the processing steps will be the same.

OPTIMAL NORMALS Computation of Optimal Normals: Additional Details Annual Updates: Simple Arithmetic Average over the period Annual Updates: Simple Arithmetic Average over the period –If more than 6 values are interpolated, the value is flagged. OCN: An N-Year Average OCN: An N-Year Average –To determine N, we compute the scaled slope and residual lag-1 autocorrelation. The latter is computed by first subtracting each time series by the Hinge Fit, yielding the residual time series. –Flagged: If 20% of are interpolated OR 20% of Hinge Fit: A Constrained Linear Fit Hinge Fit: A Constrained Linear Fit –The Hinge point is fixed at The same flag criteria as for OCN. Tuesday, June 2, 2009

OPTIMAL NORMALS Caveats The data set is different from that used to compute NOAA’s official Normals. Use care. The data set is different from that used to compute NOAA’s official Normals. Use care. Two deviations from the Livezey et al. (2007) recommendations Two deviations from the Livezey et al. (2007) recommendations –(1) The article recommends a “hybrid” approach that selects either the Hinge Fit or OCN based on certain time series characteristics. We simply provide both results individually. –(2) The authors set all negative lag-1 autocorrelations to zero. We do not. This is described further in a document called negative-lag1corr.doc Tuesday, June 2, 2009

OPTIMAL NORMALS Accessing the Products and Ancillary Files Two Options (1) Anonymous FTP: ftp://ftp.ncdc.noaa.gov/pub/data/aarguez/optimal-normals/ * This may not work if your firewall is too strict * (2) HTTP: Tuesday, June 2, 2009 Look for the readme.txt file

OPTIMAL NORMALS Directory Contents Data Files Data Files –xxx-yyy-2008.dat –“xxx” can be “ann” or “ocn” or “hin” –“yyy” can be “avg” or “max” or “min” stations.dat → station list, metadata: lat, lon, ele, name stations.dat → station list, metadata: lat, lon, ele, name Word Documents Word Documents –ams-talk-2009.doc ams-talk-2009.doc  January 2009 talk at AMS –negative-lag1corr.doc negative-lag1corr.doc  Discussion of the retention of negative residual lag1-autocorrelations Tuesday, June 2, 2009

OPTIMAL NORMALSWednesday, January 14, 2009 JUL Tmax Optimal Normals vs JAN Tmin Annual Update OCN Hinge Fit

OPTIMAL NORMALS Concluding Thoughts Optimal Normals Version 1.0 is now officially released as an experimental product. Optimal Normals Version 1.0 is now officially released as an experimental product. Please provide feedback. Please provide feedback. Please read the ‘readme.txt’ file carefully. Please read the ‘readme.txt’ file carefully. Compare to Official with care Compare to Official with care Remember: Optimal Normals are experimental products that supplement the 30-Year Normals Remember: Optimal Normals are experimental products that supplement the 30-Year Normals Tuesday, June 2, 2009

Anthony Arguez NOAA National Climatic Data Center Phone: (828) OPTIMAL NORMALS On Improving NOAA’S Climate Normals: An Introduction to ‘Optimal Normals’ of Temperature Comments or Questions? Tuesday, June 2, 2009