AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 IMPACT OF TAMDAR ON THE RUC MODEL: A LOOK INTO SOME OF THE STATISTICS WITH CASE STUDIES Ed Szoke,* Stan Benjamin, Randy Collander*, Brian Jamison*, Bill Moninger, Tom Schlatter, and Tracy Smith* NOAA/ESRL Global Systems Division Boulder, CO USA *Joint collaboration with the Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, CO

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Overview The main issue: Objective evaluation (statistics) of relative humidity (RH) has occasionally shown poorer performance for RUC runs with TAMDAR Statistics - calculated by comparing RUC forecasts with and without TAMDAR to RAOBs at the standard pressure levels (850, 700, 500 mb) Is this really worse performance with TAMDAR OR are there other reasons for the poorer scores? Procedure: Find days that stand out with poorer scores Examine individual RAOBs with forecast soundings to see where the errors occur Concentrate on the Great Lakes subset (13 RAOBs)

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Verification areas: for this study we used the inner (blue) box containing 13 RAOBs

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 RUC with TAMDAR (“dev2”, blue line) RUC run without TAMDAR (“dev”, red line) for the Great Lakes area. Bottom plot shows the difference, positive if dev2 is a better forecast than dev. Starred days highlight poorer scores for dev2. 3-h RMS error statistics for June-October 2006 at 700 mb for Great Lakes area – 13 RAOBs Dev - control Dev2 – add TAMDAR

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Statistics (RMS error) for RH for 6-h RUC forecasts valid at 0000 UTC at 700 mb for the Great Lakes area. Starred days highlight poorer scores for dev2. 6-h RMS statistics for June-October 2006 at 700 mb **** Dev - control Dev2 – add TAMDAR

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 1: 23 June 06 00z – 3h forecasts RMS score for dev2 is 7% worse than dev for 3-h forecasts valid at 0000 UTC 23 June. RAOB comparison showed 2 sites account for most of this error. Peoria, Illinois (ILX) comparison is shown here. For all plots RAOB is green, dev (RUC w/o TAMDAR) in blue dev2 (RUC w/TAMDAR) in black. The shape (character) of the dev2 RH appears to be a better match to the RAOB, but is off by ~50 mb so scores poorly at 700 mb. RH at 700 mb: RAOB = 74 % dev = 94 % dev2 = 34 % (60% error!) raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 1: 23 June 06 Pittsburgh (PIT) was the other RAOB where the RH is significantly worse for dev2 than for dev. In this case, dev2 is 39% drier than the RAOB, while dev is only 12% excessively moist. While one could argue that the shape of the dev2 RH profile better matches the changes in the vertical shown by the RAOB, the excessive drying for dev2 is probably simply not as good a forecast in this case. raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 2: 14 July 06 – 3 h 700 mb forecasts RMS scores for dev2 again were ~7% worse than for dev, for 3-h forecasts valid at 0000 UTC 14 July. RAOB comparison showed 4 sites account for most of this error. Buffalo, NY (BUF) comparison is shown here. In this case the dev2 follows the RAOB RH profile nicely until there is a more moist shift exactly at 700 mb, yielding what apperas to be an unrepresentative error at 700 mb for dev2 while dev happens to get a perfect match. RH differences: dev2 ~20% dev almost no error raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 2: 14 July 06 Aberdeen, South Dakota (ABR) comparison is shown here. The dev2 RH forecast more closely matches the RAOB up to ~770 mb, then both forecasts dry out, while the RAOB does not. While both forecasts dry at about the same rate in the vertical, it happens that the dev forecast crosses the RAOB at 700 mb. But this is because it is erroneously more moist below 750 mb! So the better score at 700 mb is not representative (with dev2 being 23% drier than the RAOB). raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 2a: 14 July 06 – 6 h 700 mb forecasts RMS scores for dev2 forecasts were ~5% worse than for dev, for 6-h forecasts valid at 0000 UTC 14 July. RAOB comparison showed about half the sites accounting for smaller errors. Buffalo, NY (BUF) 6-h comparison is shown as it illustrates the error that happens to occur with a sharp but vertically shallow more moist layer in the RAOB just at 700 mb. Nothing in other observations to know if this is real. Without this layer the dev2 forecast follows the RAOB moisture more closely than dev. raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 3: 12 October 06 – 6h forecasts valid 0000 UTC Green Bay, Wisconsin (GRB) comparison is shown here. The RMS error at 700 mb for dev2 on 12 Oct was 7% worse than for dev. Almost all of this error comes from the GRB comparison. RH & Differences at 700 mb: RAOB: 88% RH dev: 83% (-5% diff) dev2: 22% (-66% diff) The difference at 700 mb is the largest found during this period. It occurs as the dev2 forecast dries out a deep portion of the troposphere in nw flow. Is the forecast as bad as it looks? raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 3: 12 October UTC 700 mb plot The forecast from dev2 may not be as bad as it appeared. There is significant drying to the west and northwest of GRB behind the deep 700 mb upper level low. (Dewpoint is number below the temperature on the station plots.) So the main issue may be that the forecast from dev2 is just off a small amount in timing.

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 3: 12 October UTC – RAOB and dev2 comparison illustrating drying Another way to show this drying is illustrated here with an overlay of the GRB RAOB and 2 upstream RAOBs (MPX and INL), along with the dev2 6-h forecast. MPX, more to the west of GRB, is drier above 700 mb. INL, more to the nw, shows the drier layer reaching all the way down past 700 mb. Note that the dev2 forecast compares rather well to the INL RAOB, verifying nearly exactly at 700 mb.

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 4: 20 October UTC 3-h forecasts at 700 mb RMS score for dev2 is 4.5% worse than for dev for 3-h forecasts valid at 0000 UTC 20 October. RAOB comparison showed 2 sites account for most of this error (INL & MPX). International Falls (INL) comparison is shown. The deep layer of drying in the RAOB is better captured by the dev2 RH forecast, while dev appears to mainly miss this dry layer, but happens to verify better at 700 mb. RH & Differences at 700 mb: RAOB: 12% RH dev: 42% (+30% diff) dev2: 70% (+58% diff) raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 4: 20 October UTC 500 mb plot with IR Similar to the last case, drying is occurring behind a trough axis passing INL, so could argue that the character of the dev2 forecast is more representative of what is really happening than the dev forecast, though scoring worse at 700 mb.

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 5: 28 June UTC 3-h forecasts at 700 mb Better verification for dev2 (RUC w/TAMDAR) This time the RMS score for dev2 is 10% better than for dev for 3-h forecasts valid at 0000 UTC 28 June. RAOB comparisons found that a lot of variability, but some big errors for dev. Wilmington, Ohio (ILN) comparison is shown. Both forecasts have the drying beginning lower than observed, but because it does not start for dev2 until just above 700 mb it scores much better than dev. RH & Differences at 700 mb: RAOB: 78% RH dev: 13% (-55% diff) dev2: 61% (-17% diff) raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Case 6: 18 Oct UTC 3-h forecasts at 700 mb Better verification for dev2 (RUC w/TAMDAR) In this case.. dev2 is 4% better than dev (3-h forecasts ) Minneapolis, Minnesota (MSP) This case illustrates the effect of a very sharp dry layer in the RAOB (which may or may not be real). shape of both RUC forecasts is similar, but dev2 moisture profile is shifted ~30 mb lower and happens to closely match the RAOB at 700 mb, yielding an RH value 31% better than dev at 700 mb. raob dev dev2

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 What this all means... RH often varies strongly in the vertical as shown in RAOB profiles Calculating error statistics only at the mandatory levels makes them more vulnerable to unrepresentativeness It can only take 1 or 2 bad RAOB comparisons (out of 13 in the Great Lakes area) to yield a large RMS error With only mandatory levels being used, slight shifts of the RH in the vertical can be severely penalized The RAOBs often have some very sharp RH variations in the vertical that may or may not be real and can result in huge errors if they fall at a mandatory level Additionally, it is unrealistic to expect the RUC model to resolve some of these fluctuations (if they are indeed real) Considering the above reasoning, we decided to change the verification to a layer method Calculations made at 10-mb intervals

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Comparison of new scoring method with old for October 2006 Days of interest are highlighted. For 12 Oct and 20 Oct dev still scores better, but the error is much reduced (~2% RMS for layer on 12 Oct vs. 5% at 700 mb only; for 20 Oct also ~2% RMS for layer on vs. ~5% at 700 mb only). For 18 Oct, when dev2 scored better at 700 mb, the difference is also reduced by more than half. These results appear to be consistent with the sounding comparisons shown earlier. New method: 900 to 650 mb averagingOld method: 700 mb single level

AMS Annual Meeting 2007 – San Antonio 11 th IOAS-AOLS 18 January 2007 Summary We began the study as a forensic pathology study to try to better understand why the RMS RH scores were substantially worse for the RUC runs with TAMDAR on some days Used the Great Lakes area with 13 RAOB/forecast comparisons Focused on 3-h and 6-h forecasts valid at 0000 UTC since TAMDAR data in abundance for these initialization times Discovered issues with the mandatory-only scoring method Change to a layer average method have produced more representative results Found no characteristic problems with TAMDAR data or with RUC no misdesign with RUC assimilation or model additional TAMDAR in upstream airports would decrease aliasing