IHOP_2002 DATA MANAGEMENT UPDATE Steve Williams UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado 2 nd International IHOP_2002 Science Workshop Toulouse, France June 2004

IHOP-2002 Data Policy and Protocol

IHOP MOBILE FACILITY DEPLOYMENT 21 JUNE 2002 (1514 UTC)

NWS Surface Plot – 23 May 2002 (1600 UTC)

OK Mesonet Surface Plot – 23 May 2002 (1615 UTC)

IHOP_2002 Field Catalog Statistics Status Reports and Operations Summaries (600+ documents) Mission Related Reports and Products (2700+ documents) Near Real-Time Operational Image Products Satellite, Surface, Upper Air, Forecast, and Model Nearly 200,000 files/images using 20 gigabytes Report Authoring and Submission Tools Product Ingest Tools: upload forms, and ftp submissions, automated web downloading Mirror Site in Boulder, CO, with near real-time synchronization Links to other project related real-time data sources

Composites: What and Why A composite dataset is a collection (over some time period and region) of similar data (e.g. surface met) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

Composite Data Sets Developed by UCAR/JOSS for IHOP-2002 Surface Meteorological - 1 min, 5 min, and Hourly Precipitation - 15 min, Hourly, and Daily Soundings - High Resolution and 5 hPa Complete Station List for all Composites

1-min Surface Meteorological Data Composite (65 stations) ASOS (41) ARM SMOS (14) ABLE AWS (5) NCAR Supp. (5)

Oklahoma Mesonet (115) NCAR ISS/ISFF (10) West Texas Mesonet (34) 1-min sites (65) 5-min Surface Meteorological Data Composite (224 stations)

Hourly Surface Meteorological Data Composite (743 stations) CoAgMet (*) (23) AWOS (*) (110) RAWS (+) (10) SCAN (o) (6) GWMD (*) (18) Ameriflux (+) (3) HPCN (o) (72) Texas ET (o) (19) KVII-TV (+) (4) MADIS (o) (246) NMSU (*) (7) 5-min sites (o) (225)

15-minute Precipitation Composite NCDC Coop (429) 5-min surface (224)

Hourly Precipitation Composite NCDC Coop (509) NCEP hourly (1155) Hourly Met (743)

Daily Precipitation Composite NCDC Coop (1581) Hourly Met (743) NCEP Daily (1611)

ASOS AWOS ARM Others (7) OKMESO ABLE SCAN LTER HPCN WTXMESO NCAR (4) MADIS Ameriflux KVII GWMD NMSU CVT HRLY Visual MERGE Gross Limit Checks Horizontal QC Visual IHOP_2002 Hourly Surface Composite IHOP_2002 Hourly Surface Composite Development Examine Statistics CVT

Gross Limit Checking Procedures Verify file format. Data values reasonable for the Earth. Negative wind speed or precipitation values. Wind direction between 0 o and 360 o. Dew point less than or equal to air temperature. Precipitation values gross limit checked based on temporal resolution of data (i.e. allow higher precipitation rates at higher temporal resolutions).

Horizontal QC Statistical Summary Overall statistics for all data included in a composite. Summary statistics for entire networks. Summaries of the 10 worst performing stations for each network and each parameter. Summary statistics for each individual station. Summary statistics for each day. Summary statistics for each hour. Summary plots of variance values by parameter. All occurrences of zero variance noted.

What Can Lead to Surface Data Being Flagged? Instrument failure, bias, drift, random errors Incorrect latitude, longitude, or elevation Differences in siting or height of instruments Misidentified units, dates, or stations Terrain impacts Data problems at surrounding stations Unique meteorological phenomena

Example of Applied HQC Data Flags

Severe Problems Detected

Major Biases Detected

Subtle Problems Detected

Embedded Accumulative Precipitation

High Resolution and 5 hPa Radiosonde Composites Fixed Sites: Mobile Sites: NWS (14) ARM (5) ISS (1) Falcon Drops (86) Learjet (330) NCAR GLASS (146) NSSL CLASS (80)

UCAR/JOSS Atmospheric Sounding Processing Procedures Observing System Output e.g. NWS Micro-ART Convert to JOSS Quality-control Calculate Derived Format (ASCII) Parameters Vertical Consistency Checks QC Flags Gross Limit Checks Automated ChecksQC Flags e.g., u, v, dz/dt Visual Examination Error Files High-resolution and/or Interpolated Vertical Compositee.g., 5 hPa Inventory, Archive, and Develop Metadata Data and Metadata Available Online

IHOP DATA WORK IN PROGRESS ….. Transfer model output files from NOAA/FSL (e.g. new LAPS and WRF model runs) Transfer LEANDRE Lidar data and imagery files Transfer new Boundary Layer data/products Obtain OK Mesonet OASIS data and documentation Link to or transfer research radar data and imagery (PSU/NSSL) IHOP DATA MANAGEMENT – Questions – contact Steve Williams Transfer additional PI data/products Complete IHOP Instrument Intercomparison Tables

FUTURE IHOP DATA WORK? Produce CD ROM(s)? Development of IHOP GIS datasets? Design, Develop, and publish IHOP 4-D Water vapor data set? Other? IHOP DATA MANAGEMENT – Questions – contact Steve Williams