1. Project CRAFT: A Test Bed for Demonstrating the Real Time Acquisition and Acrhival of WSR-88D Base (Level II) Data Presented to UCAR/Unidata Program 12 October 2000 Kelvin K. Droegemeier Center for Analysis and Prediction of Storms and School of Meteorology University of Oklahoma

2. NEXRAD Doppler Radar Network

3. NEXRAD Facts and Figures n 158 radars (141 in the Continental US) –120 National Weather Service radars –26 Department of Defense radars –12 Federal Aviation Administration radars

4. WSR-88D Components Radar Data Acquisition

5. WSR-88D Components Radar Product Generator

6. WSR-88D Components Principal User Processor

7. WSR-88D Components Principal User Processor

8. NEXRAD Data Types n Archive Level I (analog, time domain output) n Level II data (digital base data at full resolution) n Archive Level III (“NIDS” digital products from the Radar Product Generator) n Archive Level IV (at the PUP)

9. NEXRAD Data Types n Archive Level I (analog, time domain output) n Level II data (digital base data at full resolution) n Archive Level III (“NIDS” digital products from the Radar Product Generator) n Archive Level IV (at the PUP)

10. NEXRAD Level III (NIDS) Data n 24 products available from all CONUS radars in real time n Provided by 3 commercial vendors n Lowest 4 elevation angles of the radial velocity and reflectivity –1 degree x 1 km resolution –Values are quantized (e.g., 0-5, 5-10, 10-15)

12. NEXRAD Data Types n Archive Level I (analog, time domain output) n Level II data (digital base data at full resolution) n Archive Level III (“NIDS” digital products from the Radar Product Generator) n Archive Level IV (at the PUP)

13. NEXRAD Base (Level II) Data n Full resolution digital data –Full data precision –All tilts –Not quality controlled n 1 degree x 1 km resolution for reflectivity n 1 degree by 0.25 km resolution for radial wind n Not available in real time except for selected sites (more on that later)

14. Base Data Usage: NSSL Warning Decision Support System on 3 May 1999 Courtesy National Severe Storms Laboratory

15. Trimmed Detections and Ground Truth Damage Paths Hits (142) Misses (25) FAs (21) Courtesy D. Zittel

16. 5 April 1999 - Impact of Base Data 12 Z Reflectivity Initial 700 mb Vertical Velocity Using NIDS Initial 700 mb Vertical Velocity Using Level II Data and SDVR Courtesy S. Weygandt and J. Levit

17. 5 April 1999 - Impact of Base Data 15 Z Reflectivity 3 hr ARPS CREF Forecast (9 km) Using Level II Data and SDVR Valid 15Z 3 hr ARPS CREF Forecast (9 km) Using NIDS Data Valid 15Z Courtesy S. Weygandt and J. Levit

18. Recording Level III WSR-88D Data n Limited set of products recorded on CD- ROM at the radar site

19. WSR-88D Level III Data Archival at NCDC Stations120 MediaCD Bytes/media600 MB Daily Receipts4 GB Annual Receipts60 TB

20. Courtesy Steve DelGreco (NCDC) WSR-88D Level III Data Processing

21. Level III Requests at NCDC

22. Recording WSR-88D Level II Data n Not in original plans n Exabyte 8 mm tape recorders added to all radars (1994) as an interim solution

23. WSR-88D Level II Data Archival at NCDC Stations158 Media8 mm tape* Bytes/media4.8 GB Daily Receipts165 GB Annual Receipts60 TB *6-step process with NCDC operating 92 Exabyte tape drives

24. Courtesy Steve DelGreco (NCDC) WSR-88D Level II Data Processing

25. Level II Requests at NCDC

26. WSR-88D Data Archival at NCDC % Rcpts #Sites Level II NWS64.9120 Level II DoD35.726 Level II FAA9.512 Total 88D Network55.7158 Level III NWS89.5120

27. n Currently 42,000 8-mm tapes in archive n 9,500+ copies distributed to date n Cost per tape is nominal ($100 for a radar day) but high for large (multi-year) data sets n Major time delays (2-4 months; human resources) n Archive incomplete n No mechanism for perusing the data base n NOAA spent $768,000 on Level 2 recording system repairs from 6/1/98-6/14/99 n NCDC maintains 92 Exabyte tape systems WSR-88D Base Data Archival at NCDC

28. Summary: WSR-88D Radar Data n The scientific and operational communities need base data (real time and archived) n Although NIDS data are available in real time from all WSR-88D radars, they are insufficient for many applications (NWP, hydrology) –Degradation of precision –Only the lowest 4 tilts are transmitted n Base data currently are not available in real time –Originally would have been expensive –Presumed large volume of data (10 mbytes/5 min/radar) –Need wasn’t there 10 years ago n The technology and need now exist to prototype the direct acquisition, use, and archival of base data in real time

29. The Collaborative Radar Acquisition Field Test (CRAFT) n Establish a prototype real time WSR-88D base data acquisition test bed to –Evaluate strategies for compressing and transmitting base data in real time –Develop efficient and cost-effective strategies for direct digital ingest, archive, and retrieval at NCDC –Assess the value of base data in numerical weather prediction –Test web-based data mining techniques for rapid perusal/access of base data by the scientific community

30. n CRAFT was established by the OU Center for Analysis and Prediction of Storms (CAPS) in collaboration with the –UCAR Unidata Program Office –NOAA WSR-88D Operational Support Facility –University of Washington n CAPS and the NOAA National Severe Storms Laboratory are now jointly coordinating the effort in conjunction with the NEXRAD OSF and UCAR Participants

31. RIDDS: The Foundation of CRAFT

33. Technical Strategy Repeater Hub RIDDS Linux PC Unidata LDM ($1500) Server Dedicated 56K line ($2000 - $6000/year) Cisco 1600 Series Router($2000) WSR-88D At the radar site Users Internet

34. Original CRAFT Network

35. Sharing Data from Multiple Sources via Cooperating LDMs Courtesy D. Fulker, Unidata

36. –We added to LDM in-line compression/decompression (BZIP2 off-the- shelf algorithm) n maximum 16:1 in clear air n about 8:1 in the worst case n average of about 12:1 –Maximum aggregate data rate is 15 Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for entire national network n bandwidth isn’t the issue, but rather topology, quality of service, and data storage n one year’s worth of compressed data from NWS radars is about 10 petabytes Technical Highlights

37. –Compression now being improved (hope for > 20:1) –LDM relays data to other nodes n unlimited system growth n redundancy –Simultaneous data transmission to multiple users –Single point of failure (modem is backup) Technical Highlights

38. Real Time Management and Reliability: >95% http://traffic.onenet.net/

39. Real Time Management and Reliability: >98% Steve Goodman, NASA/MSFC

40. CRAFT Phase I: Proof-of-Concept

41. n Internet2 is... –a project, led by universities, to develop advanced internet technology and applications vital to the research and education missions of higher education –big commitment n member universities have committed over $70 million per year in new investment on their own campuses n corporate members have committed more than $30 million over the life of the project Internet2 and Abilene

43. n Abilene is... –an advanced backbone network testbed n that connects regional gigaPoPs n supports Internet2 applications development n that is focused exclusively on university research and education n that is developing next generation operational and quality of service capabilities Internet2 and Abilene

44. Abilene Network January 1999 Cleveland New York Atlanta Indianapolis Kansas City Houston Denver Los Angeles Sacramento Seattle Abilene Router Node Abilene Access Node Operational January 1999 Planned 1999

45. New Concept: Abilene/Internet2 + NEXRAD

49. NCEP NCDC OU NCAR/FSL U-WA New Concept: Abilene/Internet2 + NEXRAD AWC TPC

50. Logical Network Topology Adapted from D. Fulker, UCAR

51. Alternative Strategy Repeater Hub RIDDS Linux PC Unidata LDM ($1500) Server Internet Direct Cisco 1600 Series Router($2000) WSR-88D At the radar site Users Internet

52. Abilene Backbone OtherLDMs (university, private, government) NCAR LDM NCEP LDM FSL LDM NCDC LDM OU LDM U-WA LDM Via Abilene or Commodity Internet Direct Connection to the Internet in the NWS Office Alternative Network Topology Adapted from D. Fulker, UCAR

53. –Configuration completely scalable –Everyone has real time access to WSR-88D base data (all radars) n Special quality of service for key users (NCEP, SPC) n Leverages existing infrastructure and serves as a means for developing advanced networking capabilities n Easy linkage with other networks (e.g., AWIPS, NOAAPort) n Could easily add other radars n Significant capacity for future growth (dual-pol, phased array) The Benefits

54. –Rapid and less expensive base data archival and retrieval at NCDC n Number of steps reduced to 2 n Archive time reduced by factor of 4 n Will improve archival rate (now 65%) n Reduces data request turn-around time n Reduces data volume via compression n Eliminates tape drive costs –Possible secondary back-up/real time archives where advanced data perusal and web-based data mining tools will be available The Benefits

55. n Operational reliability and redundancy in a research network n Network security n The future of the I2/Abilene network infrastructure n Coordination with regional/state networks n Capacity for growth n Phasing with deployment of the Open RPG The Challenges

56. n Use of LDM could expand significantly n How to provide LDM support to private sector users? n Integration of LDM into NEXRAD Open Systems Architecture (more later) n End-user visualization and analysis software (not just a UCAR problem) Challenges for Unidata

57. CRAFT-2 n One of 4 National Abilene Initiatives n Expand CRAFT-1 to additional 30-40 radars n Possibly include FAA radars n OU and NCAR have an opportunity to become joint online radar archive sites for data mining, visualization, etc

59. CRAFT-2 Radars Now Delivering Data May soon begin turning off some of the 8 mm recorders!!

60. WSR-88D Abilene (T1 into NCDC) NEXRAD Base Data from the 12 CRAFT Radars LDM HDSS Mass Storage At NCDC Today

61. –Web-based data mining –Analysis and visualization packages –Networking quality of service research –Digital libraries and distributed storage technology research –Storm- and meso-scale data assimilation and prediction –Mesoscale process studies –Small-scale climatologies Opportunities

62. http://geosciences.ou.edu/~jsnow/Research/Meso/

63. MS Thesis by Brad Pugh (1999) Oklahoma Climatological Survey

64. –NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC) n $540K/3 years n Research Thrusts –Test of direct ingest/archival at NCDC –Improve compression algorithms –Initial work on web-based data mining –NOAA earmark funding to OU n $474K for 1 year n Expand CRAFT to 30 radars (CRAFT-2) n Develop data assimilation capabilities for the WRF model n Kelvin doing a mini-sabbatical at the NSSL this fall –HPCC Proposal ($150K for 1 year, about to be funded) n Data mining n Network quality of service research n Hardware for additional radars Funding Status

65. The Future: FAA Radars n Initial efforts are now underway to explore extending the CRAFT infrastructure to FAA radars –TDWR (terminal Doppler weather radar) –ASR (airport surveillance radar) –ARSR (air route surveillance radar)

66. TDWR Weber (2000)

67. Airport Surveillance Radars (ASR-9) Weber (2000)

68. Airport Surveillance Radars (ASR-11) Weber (2000)

69. Air Route Surveillance Radars (ARSR-4) Weber (2000)

70. –NEXRAD is implementing an open-systems software standard based on UNIX and other widely-used technology (e.g., TCP/IP) –The temporary RIDDS system becomes the Base Data Distribution System (BDDS) –Will require moving LDM and compression into BDDS Open RPG

72. Phase I: Current Strategy Repeater Hub RIDDS Linux PC Unidata LDM ($1500) Internet Direct or 56K Phone Lines WSR-88D At the radar site Users Internet2/ Abilene

73. Phase II: BDDS + LDM/PC Repeater Hub BDDS Linux PC Unidata LDM ($1500) Internet Direct or 56K Phone Lines WSR-88D At the radar site Users Internet2/ Abilene

74. Phase III: Full BDDS Repeater Hub BDDS + LDM Internet Direct or 56K Phone Lines WSR-88D At the radar site Users Internet2/ Abilene

75. n Efforts are now underway to create a national WSR- 88D Level II (base) data ingest system n Real-time access for all users by subscribing to desired radars via LDM – LDM is a KEY component! n ESDIM project and CRAFT-2 are investigating key issues (reliability, scalability, security) n Hope to have an online digital library in Norman for data mining and research with another archive at NCAR n Pacing item for implementation is Open RPG/BDDS deployment (scheduled June ’01 – Dec ‘02) n New build for BDDS will be phased in simultaneously n Tremendous opportunities for research Summary