1. Operational Deployment of Datasets ATS 690 Jason Burks Holly Allen

2. Introduction The objective is to show the steps taken to migrate a dataset from a research facility to a weather service operational environment. The objective is to show the steps taken to migrate a dataset from a research facility to a weather service operational environment. Knowledge gained will be beneficial to both the NWS and NASA SPoRT group. Knowledge gained will be beneficial to both the NWS and NASA SPoRT group.

3. Steps Needed Identifying Dataset for Operational Use Identifying Dataset for Operational Use Display and Data Type Implementation Considerations Display and Data Type Implementation Considerations Setup of Data Ingest Setup of Data Ingest Data Reliability Data Reliability Testing of Data Testing of Data Training for Use of Dataset Training for Use of Dataset Use and Evaluation of the Dataset Use and Evaluation of the Dataset Application of the Process to Multiple Offices Application of the Process to Multiple Offices

4. Identify Dataset for Operational Use Finding Datasets Finding Datasets Dropped in lap Dropped in lap Have a Problem? Find a solution with a new dataset Have a Problem? Find a solution with a new dataset File Size is Considered File Size is Considered How much bandwidth will it require? How much bandwidth will it require? Bandwidth creates ingest limitations Bandwidth creates ingest limitations EX: ADAS datasets were pared down because of their size. EX: ADAS datasets were pared down because of their size. How much data can AWIPS hold? How much data can AWIPS hold?

5. Display and Data Type Implementation Considerations How to Display dataset in AWIPS? How to Display dataset in AWIPS? What kind of data is it? What kind of data is it? Categorized in terms of how the data will be displayed Categorized in terms of how the data will be displayed Text, Point (obs, profiler), Satellite, RADAR, Gridded (model, LMA) Text, Point (obs, profiler), Satellite, RADAR, Gridded (model, LMA) Does the data make sense? If so,… Does the data make sense? If so,… Create a color curve, contouring intervals, temporal resolution, projection information, etc. Create a color curve, contouring intervals, temporal resolution, projection information, etc. Add data to menus and volume browser. Add data to menus and volume browser. Becomes an experimental dataset. Becomes an experimental dataset. If proven needed, then determine how to get the data. If proven needed, then determine how to get the data.

6. Setup of Data Ingest How do we get the data? How do we get the data? FTP - push the data to a FTP site and go get it. FTP - push the data to a FTP site and go get it. LDM - places data in queue. Goes to whoever requests it. LDM - places data in queue. Goes to whoever requests it. From the LDAD: From the LDAD: AWIPSLDAD Firewall Regional HQ /Data/Incoming Directory Moved into a local file, where it is stored. Purged off AWIPS. A script runs from AWIPS to get files from the incoming directory on LDAD.

7. Data Reliability Once the data is ingested and displayable, is it reliable? Once the data is ingested and displayable, is it reliable? Questions to ask in order to determine if its ready for the next step: Questions to ask in order to determine if its ready for the next step: Is it timely? Is it timely? Does the data look realistic? Does the data look realistic? High quantity of erroneous data? High quantity of erroneous data? Is data consistently available? Is data consistently available?

8. Testing of Data It is hard to overcome a ‘tainted mindset’ with forecasters when they are given a bad dataset. It is hard to overcome a ‘tainted mindset’ with forecasters when they are given a bad dataset. Datasets should be tested and quality controlled before they are viewed by forecasters. Datasets should be tested and quality controlled before they are viewed by forecasters. Normally, for a dataset to be implemented, an advocate for that dataset is needed. Normally, for a dataset to be implemented, an advocate for that dataset is needed. The advocate is not a defined person, but usually someone who takes an interest in the new data, and encourages others to look at and use it. The advocate is not a defined person, but usually someone who takes an interest in the new data, and encourages others to look at and use it.

9. Training for Use of Dataset Normally is done by the Advocate or the Researcher that supplied the dataset. Normally is done by the Advocate or the Researcher that supplied the dataset. Researchers might not know how the dataset will be applied operationally, so contact with the forecaster(s) is necessary. Researchers might not know how the dataset will be applied operationally, so contact with the forecaster(s) is necessary. Contact with the Researcher is maintained in order to provide information about data outages, server problems, etc. Contact with the Researcher is maintained in order to provide information about data outages, server problems, etc.

10. Use and Evaluation of the Dataset Qualitative Qualitative Immediate feedback (i.e., surveys) Immediate feedback (i.e., surveys) Useful in identifying case studies Useful in identifying case studies Quantitative Quantitative Case study analysis. Case study analysis. Shows impact of data: Shows impact of data: Was lead time or false alarm ratio improved? Was lead time or false alarm ratio improved? In order to bring a new dataset into full operational use, there must be hard facts and numbers to back up the utility of the dataset. In order to bring a new dataset into full operational use, there must be hard facts and numbers to back up the utility of the dataset. Not: “Oh, they thought it was great.” Not: “Oh, they thought it was great.”

11. Application of the Process to Multiple Offices Color Curves Color Curves Better way to transfer between offices. Better way to transfer between offices. Recommend for future AWIPS builds an export/import option. Recommend for future AWIPS builds an export/import option. Compatibility issues between different servers. Compatibility issues between different servers. Sites with upgrades or builds vs. those not. Sites with upgrades or builds vs. those not. Allocate data key for each product Allocate data key for each product Details for each product is kept under that key. Details for each product is kept under that key. A way to apply data without getting messed up during each build process. A way to apply data without getting messed up during each build process. The extent of the dataset; if it is too large, can make it difficult to send. The extent of the dataset; if it is too large, can make it difficult to send. After dataset is provided to new offices, information on how to use/modify/troubleshoot data must also be provided. After dataset is provided to new offices, information on how to use/modify/troubleshoot data must also be provided. Written documentation. Ex: If this occurs, then check here, etc. Written documentation. Ex: If this occurs, then check here, etc. The process for implementation at the new office begins with training, whether it is done by someone in-house, someone from the previous office, or the initial research team. The process for implementation at the new office begins with training, whether it is done by someone in-house, someone from the previous office, or the initial research team.

12. Summary Identifying Dataset for Operational Use Identifying Dataset for Operational Use Display and Data Type Implementation Considerations Display and Data Type Implementation Considerations Setup of Data Ingest Setup of Data Ingest Data Reliability Data Reliability Testing of Data Testing of Data Training for Use of Dataset Training for Use of Dataset Use and Evaluation of the Dataset Use and Evaluation of the Dataset Application of the Process to Multiple Offices Application of the Process to Multiple Offices These steps should be done in order to provide a successful transfer from the research facility to operations. These steps should be done in order to provide a successful transfer from the research facility to operations.