Update on the Collaborative Radar Acquisition Field Test (CRAFT): Planning for the Next Steps Kelvin K. Droegemeier University of Oklahoma Presented to NEXRAD Technical Advisory Committee 3 December 2001 NCDC

The Collaborative Radar Acquisition Field Test (CRAFT) n A Collaborative project to demonstrate the real time compression and Internet-based transmission of WSR-88D base (Level II) data n Motivated by –Storm-scale NWP research at CAPS (now for WRF) –Decision support system development (NSSL) –Desire to improve long-term archive rate (NCDC) n Philosophy: Leverage existing infrastructures (Internet), public-domain software (Unidata LDM), and low-cost services (56K phone lines)

n CRAFT was established by the OU Center for Analysis and Prediction of Storms (CAPS) in collaboration with the –UCAR Unidata Program Office –NOAA WSR-88D Operational Support Facility (now Radar Operations Center) –University of Washington –National Severe Storms Laboratory n Now coordinated by CAPS, NSSL, ROC, Unidata, and NCDC Participants

Current Technical Strategy Repeater Hub RIDDS Linux PC + LDM + Compression ($1000) LDM Server Dedicated 56K line or ($ $6000/year) or NWS LAN Optional Router ($2000) WSR-88D At the radar site Users Internet or Abilene

n BZIP2 off-the-shelf compression algorithm –maximum 16:1 in clear air; 8:1 in the worst case; average of about 12:1 n 56K lines found to be inadequate in some cases (squall lines) – maybe 10% of time –Exceed available bandwidth by 10-20% –Can create latencies of 10s of minutes –Up to 26% improvement in compression possible in BZIP2, but requires changing code at user end –A pre-processing algorithm (NSSL) provides 10% improvement – no changes in BZIP required –56K line overrun may not be an issue – used for initial testing because of cost and security –New scan strategies and availability of cheaper, faster links (even cable modems) may render 56K lines moot Technical Highlights

Radars Now Delivering Real Time Level II Data to NCDC/OU 21

Courtesy Steve DelGreco (NCDC) Old WSR-88D Level II Data Processing at NCDC

WSR-88D Abilene T3 Compressed Level II Data From 21 Radars LDM HDSS Mass Storage At NCDC Today Modified from Steve DelGreco (NCDC) via FTP Processing/ Recompression

WSR-88D Level II Archival at NCDC % Rcpts #Sites Level II NWS Level II DoD Level II FAA9.512 Total 88D Network

WSR-88D Level II Archival at NCDC % Rcpts #Sites Level II NWS Level II DoD Level II FAA9.512 Total 88D Network CRAFT EFFICIENCY IS ABOVE 95%

n University of Iowa requested 19.4 gigabytes (uncompressed) of Level II data n The NCDC was able to retrieve it from the HDSS, compress it, and stage it to FTP in 2 hours and 5 minutes n Data was then moved over the Internet in a matter of minutes n The old 8 mm tape system would have required –58 tapes –9 workstations each running for 26 hours –A total of 232 hours n Improvement is more than a factor of 100 in time, not including personnel costs A Notable NCDC Success Story

Preliminary Latency Data

n ESDIM is funding –Analysis of latency, bandwidth, etc –Network simulation (reliability, redundancy) n Additional analysis being conducted in collaboration with the private sector Networking R&D

Additions by End of CY 2001 Cedar City Tulsa Las Vegas Flagstaff Tucson Amarillo Lubbock Oklahoma City Fort Smith Dallas/Fort Worth Wilmington Yuma Phoenix Seattle Huntsville Norfolk Charleston Morehead City Columbia Raleigh Denver Rapid City Melbourne Buffalo Chicago Pittsburgh Cleveland Cincinnati Detroit N. Indiana Cheyenne Goodland

Cedar City Tulsa Las Vegas Flagstaff Tucson Amarillo Lubbock Oklahoma City Fort Smith Dallas/Fort Worth Wilmington Yuma Phoenix Seattle Huntsville Norfolk Charleston Morehead City Columbia Raleigh Denver Rapid City Melbourne Buffalo Chicago Pittsburgh Cleveland Cincinnati Detroit N. Indiana Cheyenne Goodland Dodge City WichitaSpringfield Little Rock Shreveport Kansas City Additions by Spring, IHOP Field Program Plus CAPS FAA Research

Cedar City Tulsa Las Vegas Flagstaff Tucson Amarillo Lubbock Oklahoma City Fort Smith Dallas/Fort Worth Wilmington Yuma Phoenix Seattle Huntsville Norfolk Charleston Morehead City Columbia Raleigh Denver Rapid City Melbourne Buffalo Chicago Pittsburgh Cleveland Cincinnati Detroit N. Indiana Dodge City WichitaSpringfield Little Rock Shreveport Kansas City Cheyenne Goodland Enid Altus Military Radars

Cedar City Tulsa Las Vegas Flagstaff Tucson Amarillo Lubbock Oklahoma City Fort Smith Dallas/Fort Worth Wilmington Yuma Phoenix Seattle Huntsville Norfolk Charleston Morehead City Columbia Raleigh Denver Rapid City Melbourne Buffalo Chicago Pittsburgh Cleveland Cincinnati Detroit N. Indiana Dodge City WichitaSpringfield Little Rock Shreveport Kansas City Cheyenne Goodland Enid Altus FAA-MIT/LL Additions FY 2002 Central Ill Milwaukee Quad Cities St. Louis Paducah Louisville Charleston Baltimore Central PA Philadelphia New York City Albany Boston Binghamton 40+14=54

Current Technical Strategy Repeater Hub RIDDS Linux PC + LDM + Compression ($1000) Server Dedicated 56K line or ($ $6000/year) or NWS LAN Optional Router ($2000) WSR-88D At the radar site Users Internet or Abilene

Interim BDDS Repeater Hub BDDS Linux PC + LDM + Compression ($1000) Server Dedicated 56K lines or or NWS LAN Optional Router ($2000) WSR-88D At the radar site Users Internet or Abilene

Final BDDS Repeater Hub BDDS+LDM/ Compression Server Dedicated 56K lines or or NWS LAN WSR-88D At the radar site Users Internet or Abilene

How Does This Experiment Become an Operational System?

Abilene Network January 1999 Cleveland New York Atlanta Indianapolis Kansas City Houston Denver Los Angeles Sacramento Seattle Abilene Router Node Abilene Access Node Operational January 1999 Planned 1999

One Possible Strategy NOAA Facility via phone lines or commodity Internet Silver Spring

via phone lines or commodity Internet Abilene Network NOAA Facility Norman Boulder Silver Spring Seattle Miami Atlanta One Possible Strategy

via phone lines or commodity Internet Abilene Network NOAA Facility Norman Boulder Silver Spring Seattle Miami Atlanta “Super Sites”: Each LDM on Abilene “bus” Carries all 88D data -- redundancy One Possible Strategy

Abilene Network NOAA Facility

Abilene Network Commodity Internet NOAA Facility One Possible Strategy

Abilene Network NOAA Facility Private Sector Academia

Abilene Network NOAA Facility Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia

Abilene Network NOAA Facility Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia

Abilene Network NOAA Facility Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia Private Sector Academia

Meeting all of the Needs 88D Abilene-based network of (aggregating & relaying) LDMs, each carrying all 88D data, with redundant interconnections Private 3 Academia 2 Private 3 Academia 1 Private 4 Private 1 NOAA 3 NOAA 2 NOAA 1 Courtesy D. Fulker, UCAR

Meeting all of the Needs 88D Abilene-based network of (aggregating & relaying) LDMs, each carrying all 88D data, with redundant interconnections Private 3 Academia 2 Private 3 Academia 1 Private 4 Private 1 NOAA 3 NOAA 2 NOAA 1 Courtesy D. Fulker, UCAR

–NOAA runs its operational ingest system but links to private sector and academic systems, each providing redundancy for the other –Entire community benefits – builds upon the collaborative spirit of CRAFT –Configuration completely scalable –Leverages existing infrastructure (LDM sites, Abilene, WFO connection to commodity Internet) –Significant capacity for future growth (dual-pol, phased array) Benefits of This Concept

Weber (2000) Longer Term: Addition of FAA Radars