Project CRAFT: A Test Bed for Demonstrating the Real Time Acquisition and Acrhival of WSR-88D Base (Level II) Data Presented to UCAR/Unidata Program 12 October 2000 Kelvin K. Droegemeier Center for Analysis and Prediction of Storms and School of Meteorology University of Oklahoma

NEXRAD Doppler Radar Network

NEXRAD Facts and Figures n 158 radars (141 in the Continental US) –120 National Weather Service radars –26 Department of Defense radars –12 Federal Aviation Administration radars

WSR-88D Components Radar Data Acquisition

WSR-88D Components Radar Product Generator

WSR-88D Components Principal User Processor

WSR-88D Components Principal User Processor

NEXRAD Data Types n Archive Level I (analog, time domain output) n Level II data (digital base data at full resolution) n Archive Level III (“NIDS” digital products from the Radar Product Generator) n Archive Level IV (at the PUP)

NEXRAD Data Types n Archive Level I (analog, time domain output) n Level II data (digital base data at full resolution) n Archive Level III (“NIDS” digital products from the Radar Product Generator) n Archive Level IV (at the PUP)

NEXRAD Level III (NIDS) Data n 24 products available from all CONUS radars in real time n Provided by 3 commercial vendors n Lowest 4 elevation angles of the radial velocity and reflectivity –1 degree x 1 km resolution –Values are quantized (e.g., 0-5, 5-10, 10-15)

NEXRAD Data Types n Archive Level I (analog, time domain output) n Level II data (digital base data at full resolution) n Archive Level III (“NIDS” digital products from the Radar Product Generator) n Archive Level IV (at the PUP)

NEXRAD Base (Level II) Data n Full resolution digital data –Full data precision –All tilts –Not quality controlled n 1 degree x 1 km resolution for reflectivity n 1 degree by 0.25 km resolution for radial wind n Not available in real time except for selected sites (more on that later)

Base Data Usage: NSSL Warning Decision Support System on 3 May 1999 Courtesy National Severe Storms Laboratory

Trimmed Detections and Ground Truth Damage Paths Hits (142) Misses (25) FAs (21) Courtesy D. Zittel

5 April Impact of Base Data 12 Z Reflectivity Initial 700 mb Vertical Velocity Using NIDS Initial 700 mb Vertical Velocity Using Level II Data and SDVR Courtesy S. Weygandt and J. Levit

5 April Impact of Base Data 15 Z Reflectivity 3 hr ARPS CREF Forecast (9 km) Using Level II Data and SDVR Valid 15Z 3 hr ARPS CREF Forecast (9 km) Using NIDS Data Valid 15Z Courtesy S. Weygandt and J. Levit

Recording Level III WSR-88D Data n Limited set of products recorded on CD- ROM at the radar site

WSR-88D Level III Data Archival at NCDC Stations120 MediaCD Bytes/media600 MB Daily Receipts4 GB Annual Receipts60 TB

Courtesy Steve DelGreco (NCDC) WSR-88D Level III Data Processing

Level III Requests at NCDC

Recording WSR-88D Level II Data n Not in original plans n Exabyte 8 mm tape recorders added to all radars (1994) as an interim solution

WSR-88D Level II Data Archival at NCDC Stations158 Media8 mm tape* Bytes/media4.8 GB Daily Receipts165 GB Annual Receipts60 TB *6-step process with NCDC operating 92 Exabyte tape drives

Courtesy Steve DelGreco (NCDC) WSR-88D Level II Data Processing

Level II Requests at NCDC

WSR-88D Data Archival at NCDC % Rcpts #Sites Level II NWS Level II DoD Level II FAA9.512 Total 88D Network Level III NWS

n Currently 42,000 8-mm tapes in archive n 9,500+ copies distributed to date n Cost per tape is nominal ($100 for a radar day) but high for large (multi-year) data sets n Major time delays (2-4 months; human resources) n Archive incomplete n No mechanism for perusing the data base n NOAA spent $768,000 on Level 2 recording system repairs from 6/1/98-6/14/99 n NCDC maintains 92 Exabyte tape systems WSR-88D Base Data Archival at NCDC

Summary: WSR-88D Radar Data n The scientific and operational communities need base data (real time and archived) n Although NIDS data are available in real time from all WSR-88D radars, they are insufficient for many applications (NWP, hydrology) –Degradation of precision –Only the lowest 4 tilts are transmitted n Base data currently are not available in real time –Originally would have been expensive –Presumed large volume of data (10 mbytes/5 min/radar) –Need wasn’t there 10 years ago n The technology and need now exist to prototype the direct acquisition, use, and archival of base data in real time

The Collaborative Radar Acquisition Field Test (CRAFT) n Establish a prototype real time WSR-88D base data acquisition test bed to –Evaluate strategies for compressing and transmitting base data in real time –Develop efficient and cost-effective strategies for direct digital ingest, archive, and retrieval at NCDC –Assess the value of base data in numerical weather prediction –Test web-based data mining techniques for rapid perusal/access of base data by the scientific community

n CRAFT was established by the OU Center for Analysis and Prediction of Storms (CAPS) in collaboration with the –UCAR Unidata Program Office –NOAA WSR-88D Operational Support Facility –University of Washington n CAPS and the NOAA National Severe Storms Laboratory are now jointly coordinating the effort in conjunction with the NEXRAD OSF and UCAR Participants

RIDDS: The Foundation of CRAFT

Technical Strategy Repeater Hub RIDDS Linux PC Unidata LDM ($1500) Server Dedicated 56K line ($ $6000/year) Cisco 1600 Series Router($2000) WSR-88D At the radar site Users Internet

Original CRAFT Network

Sharing Data from Multiple Sources via Cooperating LDMs Courtesy D. Fulker, Unidata

–We added to LDM in-line compression/decompression (BZIP2 off-the- shelf algorithm) n maximum 16:1 in clear air n about 8:1 in the worst case n average of about 12:1 –Maximum aggregate data rate is 15 Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for entire national network n bandwidth isn’t the issue, but rather topology, quality of service, and data storage n one year’s worth of compressed data from NWS radars is about 10 petabytes Technical Highlights

–Compression now being improved (hope for > 20:1) –LDM relays data to other nodes n unlimited system growth n redundancy –Simultaneous data transmission to multiple users –Single point of failure (modem is backup) Technical Highlights

Real Time Management and Reliability: >95%

Real Time Management and Reliability: >98% Steve Goodman, NASA/MSFC

CRAFT Phase I: Proof-of-Concept

n Internet2 is... –a project, led by universities, to develop advanced internet technology and applications vital to the research and education missions of higher education –big commitment n member universities have committed over $70 million per year in new investment on their own campuses n corporate members have committed more than $30 million over the life of the project Internet2 and Abilene

n Abilene is... –an advanced backbone network testbed n that connects regional gigaPoPs n supports Internet2 applications development n that is focused exclusively on university research and education n that is developing next generation operational and quality of service capabilities Internet2 and Abilene

Abilene Network January 1999 Cleveland New York Atlanta Indianapolis Kansas City Houston Denver Los Angeles Sacramento Seattle Abilene Router Node Abilene Access Node Operational January 1999 Planned 1999

New Concept: Abilene/Internet2 + NEXRAD

NCEP NCDC OU NCAR/FSL U-WA New Concept: Abilene/Internet2 + NEXRAD AWC TPC

Logical Network Topology Adapted from D. Fulker, UCAR

Alternative Strategy Repeater Hub RIDDS Linux PC Unidata LDM ($1500) Server Internet Direct Cisco 1600 Series Router($2000) WSR-88D At the radar site Users Internet

Abilene Backbone OtherLDMs (university, private, government) NCAR LDM NCEP LDM FSL LDM NCDC LDM OU LDM U-WA LDM Via Abilene or Commodity Internet Direct Connection to the Internet in the NWS Office Alternative Network Topology Adapted from D. Fulker, UCAR

–Configuration completely scalable –Everyone has real time access to WSR-88D base data (all radars) n Special quality of service for key users (NCEP, SPC) n Leverages existing infrastructure and serves as a means for developing advanced networking capabilities n Easy linkage with other networks (e.g., AWIPS, NOAAPort) n Could easily add other radars n Significant capacity for future growth (dual-pol, phased array) The Benefits

–Rapid and less expensive base data archival and retrieval at NCDC n Number of steps reduced to 2 n Archive time reduced by factor of 4 n Will improve archival rate (now 65%) n Reduces data request turn-around time n Reduces data volume via compression n Eliminates tape drive costs –Possible secondary back-up/real time archives where advanced data perusal and web-based data mining tools will be available The Benefits

n Operational reliability and redundancy in a research network n Network security n The future of the I2/Abilene network infrastructure n Coordination with regional/state networks n Capacity for growth n Phasing with deployment of the Open RPG The Challenges

n Use of LDM could expand significantly n How to provide LDM support to private sector users? n Integration of LDM into NEXRAD Open Systems Architecture (more later) n End-user visualization and analysis software (not just a UCAR problem) Challenges for Unidata

CRAFT-2 n One of 4 National Abilene Initiatives n Expand CRAFT-1 to additional radars n Possibly include FAA radars n OU and NCAR have an opportunity to become joint online radar archive sites for data mining, visualization, etc

CRAFT-2 Radars Now Delivering Data May soon begin turning off some of the 8 mm recorders!!

WSR-88D Abilene (T1 into NCDC) NEXRAD Base Data from the 12 CRAFT Radars LDM HDSS Mass Storage At NCDC Today

–Web-based data mining –Analysis and visualization packages –Networking quality of service research –Digital libraries and distributed storage technology research –Storm- and meso-scale data assimilation and prediction –Mesoscale process studies –Small-scale climatologies Opportunities

MS Thesis by Brad Pugh (1999) Oklahoma Climatological Survey

–NOAA ESDIM Grant funded (CAPS+NSSL+OSF+NCDC) n $540K/3 years n Research Thrusts –Test of direct ingest/archival at NCDC –Improve compression algorithms –Initial work on web-based data mining –NOAA earmark funding to OU n $474K for 1 year n Expand CRAFT to 30 radars (CRAFT-2) n Develop data assimilation capabilities for the WRF model n Kelvin doing a mini-sabbatical at the NSSL this fall –HPCC Proposal ($150K for 1 year, about to be funded) n Data mining n Network quality of service research n Hardware for additional radars Funding Status

The Future: FAA Radars n Initial efforts are now underway to explore extending the CRAFT infrastructure to FAA radars –TDWR (terminal Doppler weather radar) –ASR (airport surveillance radar) –ARSR (air route surveillance radar)

TDWR Weber (2000)

Airport Surveillance Radars (ASR-9) Weber (2000)

Airport Surveillance Radars (ASR-11) Weber (2000)

Air Route Surveillance Radars (ARSR-4) Weber (2000)

–NEXRAD is implementing an open-systems software standard based on UNIX and other widely-used technology (e.g., TCP/IP) –The temporary RIDDS system becomes the Base Data Distribution System (BDDS) –Will require moving LDM and compression into BDDS Open RPG

Phase I: Current Strategy Repeater Hub RIDDS Linux PC Unidata LDM ($1500) Internet Direct or 56K Phone Lines WSR-88D At the radar site Users Internet2/ Abilene

Phase II: BDDS + LDM/PC Repeater Hub BDDS Linux PC Unidata LDM ($1500) Internet Direct or 56K Phone Lines WSR-88D At the radar site Users Internet2/ Abilene

Phase III: Full BDDS Repeater Hub BDDS + LDM Internet Direct or 56K Phone Lines WSR-88D At the radar site Users Internet2/ Abilene

n Efforts are now underway to create a national WSR- 88D Level II (base) data ingest system n Real-time access for all users by subscribing to desired radars via LDM – LDM is a KEY component! n ESDIM project and CRAFT-2 are investigating key issues (reliability, scalability, security) n Hope to have an online digital library in Norman for data mining and research with another archive at NCAR n Pacing item for implementation is Open RPG/BDDS deployment (scheduled June ’01 – Dec ‘02) n New build for BDDS will be phased in simultaneously n Tremendous opportunities for research Summary