1. Data Integration: Assessing the Value and Significance of New Observations and Products John Williams, NCAR Haig Iskenderian, MIT LL NASA Applied Sciences Weather Program Review Boulder, CO November 19, 2008

2. Data Integration Goals –Integrate NASA-funded research into NextGen 4-D data cube for SAS products and decision support –Evaluate potential of new data to contribute to NextGen product skill, in context of other data sources –Provide feedback on temporal/spatial scales and operationally significant scenarios where new data may contribute Approaches –Perform physically-informed transformations and forecast system integration, e.g., into fuzzy logic algorithm –Use nonlinear statistical analysis to evaluate new data importance in conjunction with other predictor fields –Implement, evaluate and tune the system

3. Example of Forecast System Integration: SATCAST integration into CoSPA

4. Numerical Forecast Models CoSPA Weather Product Generator CoSPA Situation Display NEXRADTDWR Weather Radar LLWASASOS Surface Weather Lightning Canadian Satellite Air Traffic Managers Airline Dispatch Decision Support Tools CoSPA 0-2 hour Forecasts

5. Overview of Heuristic Forecast

6. Generation of Interest Images Interest Images: –Are VIL-like (0-255) images that have a high impact upon evolution and pattern of future VIL –Result from combining individual predictor fields using expert meteorological knowledge and image processing for feature extraction

7. Creating Interest Images Convective Initiation Forecast Engine Lower Tropospheric Winds/Speed Regional CI Weights Orientation and elongation of elliptical kernel prescribed by winds Cumulus CI Interest Locations prescribed by CI Scores Stability Mask Number CI Indicators & Visible Unfavorable for CI Favorable for CI Predictor FieldsImage ProcessingFeature Extraction

8. Creating Interest Images Radar Trends and Rapid Growth Prior Image Current Image Precip & Echo Top Images Difference Images (Each Advected to Analysis Time) Trend Images Radar Boundary Growth Apply match template kernels to each pixel Precip Growth Radar Boundary Growth

9. Feature Extraction Weather Classification Line Stratiform Large Airmass Small Airmass Embedded

10. Overview of Heuristic Forecast

11. Forecast Engine Combine Interest Images  weight * Pixel Value)  weight VIL Long-term Trend Satellite Interest RADAR Boundary Weather Type Image Combined Forecast Image P (t,pixel,wxtype) = Short-term Trend.....

12. Example of VIL Interest Evolution

13. Summary of Heuristic Approach and Limitations Individual interest images are each 0-255 VIL-like images resulting from a combination of predictor fields and feature extraction Forecast is a weighted average of all interest images dependent on lead time and WxType, with weights determined heuristically –Combines static set of interest images into 0-2 hour forecasts –Storm evolution is embedded in the weights, dependent on WxType Limitations: –The process of integrating a candidate predictor is a manual, time- intensive process –The utility of the predictor or an interest image to the forecast is known only qualitatively –There may be other predictor fields and interest images that would be helpful that are not being currently used –Interest image weights and evolution functions may not be optimal –An objective method could help address these issues

14. Automated Data Importance Evaluation: Random Forests

15. Random Forest (RF) A non-linear statistical analysis technique Produces a collection of decision trees using a “training set” of predictor variables (e.g., observation and model datafeatures) and associated “truth” (e.g., future storm intensity) values –each decision tree’s forecast logic is based on a random subset of data and predictor variables, making it independent from others –during training, random forests produce estimates of predictor importance

16. Example: CoSPA combiner development (focus on 1 hour VIP level prediction) Analyzed data collected in summer 2007 –Radar, satellite, RUC model, METAR, MIT-LL feature fields, storm climatology and satellite-based land use fields –Transformations distances to VIP thresholds; channel differences disc min, max, mean, coverage over 5, 10, 20, 40 and 80-km radii –Used motion vectors to “pull back” +1 hr VIP truth data to align with analysis time data fields For each problem, randomly selected balanced sets of “true” and “false” pixels from dataset and trained RF –VIP  3 (operationally significant convection) –initiation at varying distances from existing convection Plotted ranks of each predictor (low rank is good) for various scenarios

17. VIL8bit 06/19/2007 23:30

18. VIL8bit_40kmMax 06/19/2007 23:00

19. Example fields VIL8bit_40kmPctCov 06/19/2007 23:30

20. VIL8bit_distVIPLevel6+ 06/19/2007 23:30

21. Importance summary for VIP  3 (var. WxType) Importance Rank more important less important MITLL WxType

22. Importance summary for init 10 km from existing storm Importance Rank more important less important MITLL WxType

23. Importance Rank more important less important MITLL WxType Importance summary for init 20 km from existing storm

24. Importance Rank more important less important MITLL WxType Importance Rank more important less important Importance summary for init 40 km from existing storm

25. Importance Rank more important less important MITLL WxType Importance summary for init 80 km from existing storm

26. RF Empirical Model Performance: VIP  3 Random Forest votes for VIP >= 3 Fract. Instances with VIP >= 3 Calibration ROC Curve (blue) RF empirical model provides a probabilistic forecast performance benchmark

27. Summary and Conclusions Developing satellite-based weather products may be only the first step of their integration into an operational forecast system Integration into an existing forecast system may require physically-informed transformations and heuristics An RF statistical analysis can help evaluate new candidate predictors in the context of others –Relative importance –Feedback on scales of contribution –Also supplies an empirical model benchmark Successful operational implementation may require additional funding beyond initial R&D