1. P2.5
Sensitivity of Surface Air Temperature Analyses to Background and Observation Errors
Daniel Tyndall and John Horel
Department of Atmospheric Sciences, University of Utah
Introduction
This study presents a methodology for determining overfitting and analysis accuracy using a local variational data assimilation system and a Hilbert curve in its data denial methodology
Methodology is tested using a detailed case study as well as analyses from two additional days
Case Study
Examination of 4°4° area over Shenandoah Valley, VA
RMSE values illustrate overfitting when the analysis is drawn too tightly (e.g. Exp. 7 in table) to observations:
Analyzing strong surface inversion situation: 0900 UTC 22 October 2007
Difference between RMSE at withheld and all observations becomes very large
Constraining the analysis (b and c) too much to the observations results in an unphysical temperature feature near the spine of the Blue Ridge Mtns.
Sensitivity also increases
Goal isn’t to minimize RMSE, but to minimize difference between RMSE at withheld and all observations (want similar analysis error in data voids and data rich areas)
Increasing σo2/σb2 and decorrelation length scales (e and f) results in greater lateral and vertical influence of the observations
RMSE and Sensitivity – 0900 UTC 22 October 2007 #
Experiment
RMSE Using All Observations (°C)
RMSE Using Withheld Observations (°C)
Sensitivity (°C) B
Background
2.15 - 1
R = 40 km, Z = 100 m, σo2/σb2 = 1
1.62
1.93
0.26 2
R = 80 km, Z = 200 m, σo2/σb2 = 1
1.80
1.98
0.29 3
R = 20 km, Z = 50 m, σo2/σb2 = 1
1.41
1.89
0.20 4
R = 40 km, Z = 100 m, σo2/σb2 = 0.5
1.54
1.94
0.34 5
R = 40 km, Z = 100 m, σo2/σb2 = 2
1.70
0.19 6
R = 80 km, Z = 200 m, σo2/σb2 = 2
1.83
0.22 7
R = 20 km, Z = 50 m, σo2/σb2 = 0.5
1.67
1.90
Local Surface Analysis (LSA)
2D variational surface analysis system that utilizes downscaled Rapid Update Cycle (RUC) 1-hr forecast for its background
Unphysical features can develop in background field during temperature inversions from downscaling
2-m air temperature observations assimilated from various mesonets and METAR sites
Observations must fall within a ±12 min time window about analysis hour; or −30/+12 min time window for RAWS networks
Background error covariance specified using background error variance horizontal and vertical decorrelation length scales
Further Evaluation
Two additional days of analyses analyzed to further evaluate method
20 May 2009 – synoptically quiescent period with morning inversion
26 May 2009 – synoptically active period with regressing and progressing front through domain
Additional analyses confirm difference between RMSE values at all and withheld observations increases when analysis is constrained too tightly
Sensitivity across grid also increases (compared to other experiments) b. c. d. f. e. a.
Washington, D.C.
Blue Ridge Mtns.
Shenandoah Valley
Appalachian Mtns.
Analysis, R = 40 km, Z = 100 m, σo2/σb2 = 1
Increments, R = 40 km, Z = 100 m, σo2/σb2 = 1
Analysis, R = 80 km, Z =200 m, σo2/σb2 = 2
Increments, R = 80 km, Z = 200 m, σo2/σb2 = 2
Background
Domain Topography
Domain Topography
Analysis, R = 40 km, Z = 100 m, σo2/σb2 = 1
Increments, R = 40 km, Z = 100 m, σo2/σb2 = 1
Background
Analysis, R = 80 km, Z =200 m, σo2/σb2 = 2
Increments, R = 80 km, Z = 200 m, σo2/σb2 = 2
RMSE and Sensitivity – 20 May 2009 – Synoptically quiescent period #
Experiment
RMSE Using All Observations (°C)
RMSE Using Withheld Observations (°C)
Sensitivity (°C) B
Background
2.41
2..41 - 1
R = 40 km, Z = 100 m, σo2/σb2 = 1
1.99
2.23
0.24 2
R = 80 km, Z = 200 m, σo2/σb2 = 1
2.12
2.22
0.19 3
R = 20 km, Z = 50 m, σo2/σb2 = 1
1.81
2.25
0.23 4
R = 40 km, Z = 100 m, σo2/σb2 = 0.5
1.94
0.30 5
R = 40 km, Z = 100 m, σo2/σb2 = 2
2.04
0.18 6
R = 80 km, Z = 200 m, σo2/σb2 = 2
2.14
0.16 7
R = 20 km, Z = 50 m, σo2/σb2 = 0.5
1.72
2.28
0.31
Covariance Estimation
Original σo2:σb2 suggested for background field was 1:1, with horizontal (R) and vertical (Z) decorrelation length scales of 40 km and 100 m respectively
Covariances determined through a statistical analysis on month-long sample of observations across CONUS (see Myrick and Horel (2006) for details on technique)
Results of statistical analysis show σo2:σb2 should be doubled (2:1)
Observation errors remain strongly correlated beyond original decorrelation length scales
RMSE and Sensitivity – 26 May 2009 – Synoptically active period #
Experiment
RMSE Using All Observations (°C)
RMSE Using Withheld Observations (°C)
Sensitivity (°C) B
Background
2.25 - 1
R = 40 km, Z = 100 m, σo2/σb2 = 1
1.81
2.01
0.21 2
R = 80 km, Z = 200 m, σo2/σb2 = 1
1.92
0.17 3
R = 20 km, Z = 50 m, σo2/σb2 = 1
1.65
2.03 4
R = 40 km, Z = 100 m, σo2/σb2 = 0.5
1.77
0.27 5
R = 40 km, Z = 100 m, σo2/σb2 = 2
1.85
1.93
0.16 6
R = 80 km, Z = 200 m, σo2/σb2 = 2
1.94
0.14 7
R = 20 km, Z = 50 m, σo2/σb2 = 0.5
1.58
2.06
Data Denial Methodology
Observations were randomly withheld using the Hilbert curve
Hilbert curve uniformly removes observations from non-randomly distributed observation networks
Two error measures computed:
Root-mean-square error (RMSE) at observation gridpoints:
Root-mean-square sensitivity at all gridpoints:
Summary
Overfitting occurs when analysis is drawn too tightly to observations; results in large difference between RMSE computed at all locations compared to that computed at withheld locations
Preferable to have analyses with comparable errors in data rich regions relative to those in data void areas
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
R = 40 km, Z =100 m
R = 80 km, Z =200 m
Using the Hilbert curve to withhold observations. (1) The analysis domain is converted into a unit square. (2) The unit square is subdivided over and over again into smaller squares until there is a maximum of one observation in each of the smaller squares (some squares may be empty as well). (3) The Hilbert curve is drawn through the domain, with each vertex of the curve occupying each of the smaller squares. (4) Vertices of the Hilbert curve without observations are condensed down and removed from the curve. (5) Observations are withheld based on the order they are located on the curve.