1. The Collaborative Radar Acquisition Field Test (CRAFT): A Unique Public- Private Partnership in Mission-Critical Data Distribution Kelvin K. Droegemeier University of Oklahoma Internet2 Spring Meeting 20 April 2004 NCDC

2. NEXRAD Doppler Radar Network

3. The Moore, Oklahoma Tornado of 3 May 1999

4. Storm A Storm B Storm D Storm E Storm C Storm H Storm G 100 km range ring Damage Paths Ground Truth - 53 Tornadoes Courtesy D. Zittel

5. Trimmed Detections and Ground Truth Damage Paths Hits (142) Misses (25) FAs (21) Courtesy D. Zittel

6. USA Today Doppler Weather Radar: Measuring the Wind Speed

7. A Unique Tool for Sampling the 3D Atmosphere in Detail n The radar scans a 3-D conical volume of the atmosphere every 5-6 minutes through a series of 360 degree sweeps…

8. n … at several elevation angles A Unique Tool for Sampling the 3D Atmosphere in Detail

10. NEXRAD Facts and Figures n 159 radars (142 in the Continental US) –121 National Weather Service radars –26 Department of Defense radars –12 Federal Aviation Administration radars

11. NEXRAD Data Types n Archive Level I –analog, time domain output n Level II data –digital base data at full precision and all elevation angles n Archive Level III –digital products + lowest four elevation angles –reduced precision: data are binned n Archive Level IV

12. n Archive Level I –analog, time domain output n Level II data –digital base data at full precision and all elevation angles n Archive Level III –digital products + lowest four elevation angles –reduced precision: data are binned n Archive Level IV NEXRAD Data Types

13. The Value – Hazard Detection

14. The Value – Real Time 3D Analyses http://caps.ou.edu

15. The Value – Storm Climatologies

16. Carbone et al. (2002) The Value – Fundamental Predictability

17. The Value – Computer Prediction Models

18. 6 pm 7 pm8 pm Radar Hourly Radar Observations (Fort Worth Shown by the Pink Star)

19. 6 pm 7 pm8 pm Radar Computer Forecast 2 hr 3 hr 4 hr

20. Fcst w/o Radar 2 hr 3 hr 4 hr 6 pm 7 pm8 pm Radar

21. Space Shuttle Columbia Debris Trail

22. Bird Migration Studies

23. NEXRAD data are *THE* most valuable data for mesoscale meteorology… but they were not available in real time to the academic and corporate communities

24. The Collaborative Radar Acquisition Field Test (CRAFT) n Begun in fall, 1998 as a collaborative R&D project to demonstrate the real time compression and Internet-based transmission of WSR-88D base (Level II) data n Philosophy: Leverage existing infrastructures (Internet), public-domain software (Unidata LDM), and secure, low-cost services (56K phone lines) n Short-term goals: Storm-scale research and improvement of NCDC archive capability n Long-term goal: National real-time capability for operations, research, and private industry

25. n Initiated and led by the OU Center for Analysis and Prediction of Storms (CAPS) n National Severe Storms Laboratory (NSSL) n NOAA WSR-88D Radar Operations Center (ROC) n UCAR Unidata Program n National Climatic Data Center (NCDC) n University of Washington TREMENDOUS support from the US Weather Research Program Coordinating Organizations

26. Foundation #1: NSSL’s RIDDS Courtesy NSSL

27. Foundation #2: Unidata Local Data Manager (LDM) Courtesy D. Fulker, Unidata

28. Foundation #3: Data Compression – BZIP2 The Internet Adaptation to LDM by Harry Edmon, U. of Washington

29. Foundation #4: The Internet The Internet Invention courtesy A. Gore Commodity NGI Internet2 Abilene

30. Foundation #5 n Incredible cooperation, collegiality, and agency support NCDC

31. Original Technical Strategy RIDDS WSR-88D At the radar site

32. Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) WSR-88D At the radar site

33. Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) Router ($2000) WSR-88D At the radar site

34. Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) Dedicated 56K line, T1, ($2000 - $6000/year) or NWS LAN Router ($2000) WSR-88D At the radar site LDM Server (OU)

35. Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) LDM Server (OU) Dedicated 56K line, T1, ($2000 - $6000/year) or NWS LAN Router ($2000) WSR-88D At the radar site Users (LDM) Internet or Abilene

36. n BZIP2 off-the-shelf compression algorithm –maximum 16:1 in clear air; 8:1 in the worst case; average of about 12:1 n Maximum aggregate data rate is 15 Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for entire national network –bandwidth isn’t the main issue, but rather topology, quality of service, and data storage n 56K lines found to be inadequate in some cases (squall lines) – maybe 10% of time –Exceed available bandwidth by 10-20% –Can create latencies of 10s of minutes (more later) –Up to 26% improvement in compression possible in BZIP2, but requires changing code at user end –A pre-processing algorithm (NSSL) provides 10% improvement – no changes in BZIP required Technical Highlights

37. Humble Beginnings… January, 1999

38. Slow But Sure Growth February, 2000

39. Eight More in 1 Year January, 2001 14 Radars

40. Latency Data

41. The Idea Catches Hold January, 2002

42. The NWS Makes a Bold Decision Fall 2004

43. Regional Collection Concept Proposed 5 Yrs Ago NCEP NCDC

44. Reiterated Fall 2002

45. n NWS is implementing a data delivery system based upon the CRAFT concept (regional hubs, Abilene as the backbone) n This infrastructure could become THE NWS data and model grid delivery system of the future Current Status

47. We’ve Come a Long Way! Today – 123 Radars!!

48. –Initiated after 2001 Stakeholders Workshop –NWS agreed to let OU provide real time Level II data to private companies and others (academia), as R&D partners, in exchange for latency and other information to help evaluate system performance –Any group could participate –Provided an opportunity for private sector partners to gain experience using Level II data and develop new capabilities with them –Had 6 private sector partners –Companies agreed to the collaborative nature of the project, i.e., everyone wins when everyone contributes Private Sector Joins the Effort

49. The Benefits – Industry Development Baron Services, Inc.

50. A True Public-Private Partnership Recent NOAA + Pvt Sector Money  $3.0M

51. n 123 NEXRAD radars are delivering Level II data to University of Oklahoma in real time –Subsequent distribution to more than 30 users including NCDC, NWS HQ, UCAR Unidata, numerous universities and NOAA laboratories, and several (paying) private companies n Extremely high reliability (99+%) with latency less than 10 sec n Top-tier nodes announced during week of 12 April 2004 –University of Oklahoma –Purdue University –ERC Current Status

53. IRaDS will continue to collect and analyze network information for R&D

58. Data Ingest is Sized to Handle Future NEXARD Improvements Enhancement Factor of Data Flow Increase (Relative to August 2002) Possible Implementation Date Cumulative Data Flow Increase (Relative to August 2002) 4.1 min VCP 1.24 Jan 2003 1.24 TCP/IP Wideband Interface 1.00 July 2004 1.24 0.25 km Reflectivity Data 1.56 October 2004 1.93 0.5 deg Azimuth Sampling 2.00 January 2005 3.86 Doppler Data to End of 2 nd Trip 1.49 October 2005 5.76 Clutter Filters and SNR Thrshld 1.97 January 2005 11.35 Dual Polarization 2.32 January 2008 26.35

59. n OU and NSSL have worked with FAA Mike Monroney Aeronautical Center in OKC to transmit TDWR data in real time n Can be extended to all TDWR, ASR and ARSR systems n The idea of using Abilene to transmit data was suggested by Lincoln Laboratory 2 years after CRAFT was underway Weber (2000) Addition of FAA Air Traffic Radars

60. n Baron Services centrally collects data now (Huntsville, AL) n TV stations only scan lowest elevation angle, which would be very useful in biological target ID and tracking Possible Addition of TV Station Radars Baron Services TV Radar Sites

61. Addition of Experimental Data from the Next NEXRAD n Multi-institutional effort led by NSSL n Spy-1 military radar technology n Totally solid state; can track aircraft simultaneously n Test bed is now up and running in Norman

62. New NSF Engineering Research Center for Adaptive Sensing of the Atmosphere (CASA) n UMass/Amherst, OU, CSU, UPRM n Concept: inexpensive, phased array Doppler radars on cell towers and buildings n Adaptive dynamic sensing of multiple targets including aircraft

63. Courtesy Steve DelGreco (NCDC) Old WSR-88D Level II Data Processing at NCDC

64. WSR-88D Level II Archival at NCDC % Rcpts #Sites Level II NWS64.9120 Level II DoD35.726 Level II FAA9.512 Total 88D Network55.7158 CRAFT Efficiency is ABOVE 95%

65. Old Use of NCDC Archived Level II Radar Data Courtesy Steve DelGreco (NCDC)

66. WSR-88D Abilene T3 Compressed Level II Data From 123 Radars LDM HDSS Mass Storage At NCDC Today Modified from Steve DelGreco (NCDC) via FTP Processing/ Recompression

67. Stats from NCDC: 8/1/02-9/23/02 Courtesy S. Delgreco

69. Thank You Abilene and Internet2!!!