Implementation and Plans for TIGGE at NCAR and ECMWF Douglas Schuster, Steven Worley, Nathan Wilhelmi NCAR Baudouin Raoult, Manuel Fuentes, Jorg Urban ECMWF 4/27/2019

Archive Center Data Collection Outline Archive Center Data Collection Data Discovery, Access, And Distribution Analysis Tools Future User Services Summary 4/27/2019

Archive Center Data Collection Participating Archive Centers National Center for Atmospheric Research (NCAR) European Center for Medium Range Weather Forecasts (ECMWF) China Meteorological Administration (CMA) Data Collection Mechanism Unidata’s Internet Data Distribution/Local Data Manager system (IDD/LDM) History of providing similar functionality in delivering National Center for Environmental Prediction (NCEP) model data to the university community. 4/27/2019

Archive Center Data Collection Current Data Providers Future Data Providers Australia China Canada Brazil Korea France Center Startdate ECMWF 10/1/2006 United Kingdom (UKMO) Japan (JMA) NCEP 11/1/2006 4/27/2019

Archive Center Data Collection TIGGE Parameters Available on 4 level types Single Level (includes surface) Pressure Level (1000, 925, 850, 700, 500, 300, 250, 200, 50 hPa) 50 hPa level only includes geopotential height Isentropic Level (320K) Potential Vorticity Level (2 PVU) 4/27/2019

Archive Center Data Collection TIGGE Archive at NCAR Model output stored as forecast files for each data provider Each file contains all parameters and ensemble members for a level type and forecast time Complete archive maintained on the Mass Store System as part of the CISL Research Data Archive (RDA). TIGGE Archive at ECMWF Data archived through ECMWF’s MARS system. Stores individual GRIB messages. 4/27/2019

Archive Center Data Collection Summary of Data Providers 4/27/2019

Archive Center Data Collection Pressure Level Parameters 4/27/2019

Archive Center Data Collection Isentropic Level Parameters Potential Vorticity Level Parameters 4/27/2019

Archive Center Data Collection Single Level Parameters 4/27/2019

Archive Center Data Collection Single Level Parameters cont. 4/27/2019

Data Discovery, Access, and Distribution NCAR’s TIGGE Web Portal (http://tigge.ucar.edu) Users can search, discover, and download forecast files Select data by initialization date/time, data provider, parameter level, and forecast time Each file contains all parameters and ensemble members for a given level type (sl, pl, pv, pt) Contains most recent 2-3 weeks of model output 48 hour data access delay from model init time 4/27/2019

Data Discovery Access and Distribution http://tigge.ucar.edu 4/27/2019

Data Discovery Access and Distribution http://tigge.ucar.edu 4/27/2019

Data Discovery Access and Distribution NCAR Research Data Archive http://dss.ucar.edu/datasets/ds330.0/ -2006 http://dss.ucar.edu/datasets/ds330.1/ -2007 The RDA dataset enables: Access to the complete TIGGE archive. Easy use of the TIGGE archive on NCAR CISL computers – data coming directly from the Mass Store System (MSS) NCAR computing accounts available upon request Support staff to handle requests for offline data Forecast file structure identical to online files 4/27/2019

Data Discovery Access and Distribution http://dss.ucar.edu/datasets/ds330.0 4/27/2019

Data Discovery Access and Distribution http://dss.ucar.edu/datasets/ds330.0 4/27/2019

4/27/2019

Analysis Tools User Analysis and Basic Data Manipulation Tools WMO GRIB2 is relatively new, so tools are immature Forecasts with ensemble members add another dimension Improvements underway in: NCAR Command Language (NCL)/Python GEMPACK, GRIB-Java (Unidata) NOAA tools (wgrib2, GRIB-2 software libraries, etc) ECMWF GRIB API http://tigge.ucar.edu/data-tools.htm 4/27/2019

Future User Services TIGGE Portal 2007 Improvements Develop streaming download for multiple files Upgrade to handle subset data requests through a simple interface (e.g. parameter selection) Provide user selected grid interpolation across multiple models Add spatial sub-setting functionality Include subscription services for recurring requests Provide web services for automated requests Common interface for NCAR, ECMWF, and CMA 4/27/2019

Summary ECMWF and NCAR now archiving data from 4 providers (ECMWF, NCEP, UKMO, JMA) 6 more data providers plan to come online over 2007 (Australia, Brazil, Canada, China, Korea, France) The TIGGE archive and access system designed to accommodate irregularity between providers. Desired 2007 Improvements include: Streaming download for multiple forecast files. User specified sub-set/grid interpolation requests across multiple models. A common web service interface for CMA, ECMWF, and NCAR for automated requests. 4/27/2019

TIGGE Portal / Web Services Architecture Java 5 based implementation. Developed with the Spring application framework. Web Portal Interface Point and click interface allows the selection and downloading of forecast dataset. Web Services REST based Web Services to allow clients to discover and download data. Provides an interface for automated clients to download data in large volumes and at regular intervals. 4/27/2019

REST = Representational State Transfer. REST Web Services REST = Representational State Transfer. REST is not a standard for web services, it is a style of web service. REST is built on existing standards (HTTP/URL/XML). Requests are sent as valid HTTP requests using the appropriate verbs (GET, POST, PUT, DELETE). Responses are returned service specific XML documents. Since REST is not a standard it avoids some of the pitfalls and problems associated with SOAP based web service interoperability. In this particular application it allows for returning complex data types that are not inherently bound to implementation toolkit/language. 4/27/2019

HTTP Request/Response HTTP/XML Rest Request/Response TIGGE Data Selection/Download Architecture Tomcat Engine MySql TIGGE Forecast Metadata Database Browser Client HTTP Request/Response TIGGE Web Portal Interface Forecast Metadate Database Access Forecast Catalog TIGGE Service Web Service Client HTTP/XML Rest Request/Response TIGGE Web Service Interface GRIB2 Forecast Data Files 4/27/2019

HTTP Request/Response HTTP/XML Rest Request/Response TIGGE Data Subsetting Architecture Tomcat Engine MySql TIGGE Subset Request Database Browser Client HTTP Request/Response TIGGE Web Portal Interface Forecast Metadate Database Access Forecast Catalog TIGGE Service Web Service Client HTTP/XML Rest Request/Response TIGGE Web Service Interface Perl Server Embedded HTTP Client Data Extraction Engine 4/27/2019

Archive Center Data Collection Current TIGGE Data Ingest Status Data is coming from three separate IDD/LDM systems ECMWF (Data Provider = ECMWF, UKMO, JMA) Overwhelmingly the largest CONDUIT (NCEP), maintained by Unidata CPTEC (Brazil) ECMWF (UKMO) JMA CPTEC NCEP Unidata Server NCAR 4/27/2019

Archive Center Data Collection Hourly Volume Receipt from IDD EXP Stream 4/27/2019

Archive Center Data Collection Hourly Volume Receipt from IDD CONDUIT Stream 4/27/2019

Archive Center Data Collection Hourly Cumulative Volume Data Summary 4/27/2019

Archive Center Data Collection Highlights Succeeded to reach target data rate (10 GB/hr) – Oct. 2005 Build up supporting systems and software around IDD/LDM Develop archiving procedures and implement file management for the TIGGE portal. Begin realistic daily test data flow – April 2006 Initiate operational data collection October 1, 2006 4/27/2019

Sub-setting and Grid Interpolation Future User Services Sub-setting and Grid Interpolation Users will need more than file downloads at model center native resolution Specify parameter, temporal, and spatial sub-setting across all models Requires verified software from each data provider Must build and run effectively on local hardware 4/27/2019

TIGGE - Who, What, Why? WMO THORPEX Interactive Grand Global Ensemble (TIGGE) Foster multi-model ensemble studies. Improve accuracy of 1 to 14 day high-impact weather forecasts. Up to 10 operational centers contributing ensemble data. Two Phases 1: Three central Archive Centers China Meteorological Administration (CMA) European Center for Medium-Range Weather Forecasts (ECMWF) National Center for Atmospheric Research (NCAR) 2: Widely Distributed Access (not discussed today) 4/27/2019

Archive Center Data Collection Cooperative Support Team for Data Ingest and Highlights ECMWF – Raoult, Fuentes Optimally tune ECWMF and NCAR systems to work together Develop protocols to ensure complete transfer (e.g. grid manifests) Unidata – Yoksas Expert advise on IDD/LDM VETS – Brown Installed, configured/re-configured, monitored IDD/LDM DSG – Arnold System administration NETS – Mitchell TCP packet analysis and advise 4/27/2019

Current and Future Challenges Transition from Dataportal to Ultra-zone was not seamless Machine differences perturbed a number of adjustments Clock synchronization with ECMWF TCP Cache, and File IO settings LDM configuration Things are running well now. 4/27/2019

Data transport by Unidata’s IDD/LDM application TIGGE Status @ NCAR Data Received Data transport by Unidata’s IDD/LDM application Tuned to a maximum of about 10 GB/Hour (optimum rate between ECMWF and NCAR) Transfer packets are individual GRIB2 messages (single 2D fields) Current receipt metrics, 172 GB/Day, 809K fields Data organization and storage Fields are combined into forecast files organized by: Initialization time, data provider, level type, and forecast time step All ensemble members are included in each forecast file Most current three-week period kept online (4-6TB) Long-term times series archived on the NCAR Mass Storage System This file based approach contrasts with the ECMWF MARS approach Optimum rate => no data loss and no need, in general, for resend requests between ECMWF and NCAR Level type = single level, pressure level, potential vorticity level, and potential temperature level 4/27/2019

Current participation TIGGE Status @ NCAR Data Providers Current participation * N200, Reduced Gaussian There are 71 ‘standard’ grids, varying levels of compliance now Grids = 26 surface or single level + 45 pressure level Future contributors Australia, Brazil, Canada, China, Korea, and France Archive System is designed to handle varying forecasts per day, output resolution, and numbers of ensemble members Data Provider Compliant Grids Forecasts/Day (Resolution) Ensemble Members ECMWF 70 2 (N200*) 51 UK Met Office 62 2 (1.25x2/3˚) 24 JMA 48 1 (1.25x1.25˚) NCEP 41 4 (1.0x1.0˚) 15 Compliant Grids = The Archive Centers are prepare for non-full compliant starts and growing compliance with time. There are few supplementary grids, e.g. U, V, T on PV surface and PV on 320K isentropic surface 4/27/2019

Design and implement user registration TIGGE Status @ NCAR Next Steps Design and implement user registration Simple online form, agree to research and education work only Default, 48 hour delay (from forecast init. time) before access Real-time access granted by International Program Office E.g. Special field project support Open user access Initially, only file download through a portal interface http://tigge.ucar.edu (to be activated early November) User Software, a challenge because GRIB2 is new Put examples and resource pointers into portal (end November) Improvements will be posted as they become available User defined subsets and regridding across multiple models Development depends on securing additional funding Bring new Data Providers online as they become ready 4/27/2019

Data Discovery Access and Distribution 4/27/2019

Archive Center Data Collection Summary of Data Provider Forecasts Center- Model pf Cf fc Len. (h) pl inc pt inc pv inc sl inc Daily Init Times ecmf-glob 50 1 240 6 00,12 246 – 360 egrr-glob 23 rjtd-glob 216 12 6* kwbc-glob 14 384 00,06,12, 18 4/27/2019