The International TIGGE Archive Baudouin Raoult, Manuel Fuentes - ECMWF Steven Worley, Doug Schuster - NCAR Bian Xiaofeng, Li Xiang - CMA Many Others! 3rd Thorpex Int. Science Symp., May 2009

Background: the TIGGE database Global ensemble forecasts to around 14 days generated routinely at different centres around the world Three archive centres: CMA, NCAR and ECMWF Ten data providers: – ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil) Data exchanges using UNIDATA LDM, HTTP and FTP 230 TB, 1.5 billion fields, growing by > 1 TB/week, 1.7 million fields/day

Background, Brief Time Line – March 2005, TIGGE Workshop at ECMWF Collect community input for TIGGE science and data requirements Choose centralized archives for early start (ECMWF, NCAR, CMA) – September 2005, meeting of technical experts at ECMWF Representatives from ECMWF, NCAR, CMA, and NAEFS Create technical implementation plan to address TIGGE needs – November 2005, TIGGE Working Group at ECMWF Present and get approval for the technical implementation plan – October 2006, first EPS data is delivered – May 2006, User access to TIGGE archive starts – February 2008, all 10 international providers sending EPS data Significant accomplishment, create 24x7 international system in less than three years 3rd Thorpex Int. Science Symp., May 2009

Quality assurance: homogeneity Homogeneity is paramount for TIGGE to succeed – The more consistent the archive the easier it will be to develop applications There are three aspects to homogeneity: – Common terminology (parameter names, file names,…) – Common data format (format, units, …) – Definition of an agreed list of products (Parameters, Steps, levels, …) What is not homogeneous: – Resolution – Base time (although most provider have a run at 12 UTC) – Forecast length – Number of ensemble

Strong governance Precise definition of: – Which products: list of parameters, levels, steps, units,… – Which format: GRIB2 – Which transport protocol: UNIDATAs LDM – Which naming convention: WMO file name convention Only exception: the grid and resolution – Choice of the data provider. Data provider to provide interpolation to regular lat/lon – Best possible model output Many tools and examples: – Comprehensive documentation – Sample dataset available – Various GRIB2 tools, tigge_check validator, … – Scripts that implement exchange protocol

QA: Checking for homogeneity E.g. looking for wrong units?

Completeness The objective is to have 100% complete datasets at the Archive Centres Completeness may not be achieved for two reasons: – The transfer of the data to the Archive Centre fails – Operational activities at a data provider are interrupted and back filling past runs is impractical Incomplete datasets are often very difficult to use Most of the current tools used for ensemble forecasts assume a fixed number of members from day to day

QA: Checking completeness (TIGGE history)

Archive Status and Monitoring, Variability between providers

Archive Status and Monitoring, Archive Completeness PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level VariableLvLECWFUKMOJMANCEPCMACMCBOMMetFKMACPTC Geopotential ZPL Specific HPL T U-velocityPL V-velocityPL Potential VorPT Potential TPV U-velocityPV V-VelocityPV U 10mSL V 10mSL CAPESL Conv. Inhib.SL Land-seaSL Mean SLPSL Orog.SL Skin TSL Snow D. H20SL Snow F. H20SL

Archive Status and Monitoring, Archive Completeness VariableLvLECWFUKMOJMANCEPCMACMCBOMMetFKMACPTC Soil Moist.SL Soil TSL Sunshine D.SL Surf. DPTSL Surf. ATmaxSL Surf. ATminSL Surf. ATSL Surf. PSL LW Rad. OutSL LH fluxSL Net RadSL Net Therm. RadSL Sensible Rad.SL Cloud CovSL Column WaterSL PrecipitationSL Wilt. PointSL Field Cap.SL PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

Access to TIGGE data Data is available to Research and Education after 48 hours – Self registration by agreeing to the terms and conditions Portals at CMA, NCAR and ECMWF

User Data Access and ECMWF TIGGE web site – Documentation – Model changes ECMWF portal offers: – Access to offline data – Aggregation along any axis (date, level, parameter, origin, ensemble, …) – Provision of multi-model data on a single grid (regridding to any lat/lon grid) – Sub-area selection – Reduces volumes to be downloaded by many order of magnitude

ECMWF

TIGGE portal at ECMWF (using Ajax)

TIGGE Portal: grid selection

TIGGE Portal: area selection

ECMWF

ECMWF: Scheduling users requests Monitoring Per user limits Global limits

ECMWF Batch access (recommended) Perl or Python scripts

direct access to offline data (MARS)

User Data Access and Support, NCAR User data selection and subsetting equivalent to ECMWF 2.5 Weeks of Data Online Archive organized in file groups by provider and forecast initialization time – Pressure Level, Θ Level, PV Level, Single/Surface Level Multiple file download option – Direct through browser, using prepared wget script Parameter download available through curl scripts – Provides mechanism to build recurring requests Long-term file groups available by request 3rd Thorpex Int. Science Symp., May 2009

User Data Access and Support, NCAR Subset data format in GRIB2 or netCDF Online queue to monitor subset data processing progress Subsetting completion notification for users TIGGE Research Support – Library listing of software tools for data access and analysis – TIGGE Users Group web forum Community web space to share questions and findings Provided by Unidata Archive of Tropical Cyclone (cxml) data for participating providers – Updated 6-hourly 3rd Thorpex Int. Science Symp., May 2009

User Data Access and Support, CMA 3rd Thorpex Int. Science Symp., May 2009 Collaborated with ECMWF and CNIC Grid Middleware – Accessing the distributed TIGGE data based on Grid middleware (GRIA, GOS) technologies Data retrieval and Derived Production Generation Request Monitoring and Query Access Control

User Data Access and Support, CMA Data Retrieval – Data Selection Selecting different data attributes, such as origin centre, date, base time, forecast time step, forecast type, level type, level and parameter. The different data options will be updated dynamically based on the user selection and on the constraints described in the data catalogue. – Submit data request If the number of requested fields is larger than the allowed limit, the portal will suggest the users to split their data retrieval. – Custom result products Area selection Grid conversion Plotting – Browse and download result products

User Data Access and Support, CMA Production Generation (e.g. ensemble mean) – Product selection Selecting the product and associated parameters based on the product catalogue Submit product generation request – In development, more products later Request Monitoring and Query – Check the real-time updates of the information about requests – Access the detailed monitoring information of requests for a period of time, including request creation time, request finish time, process status, request condition, detailed processing information and result download. – Query monitoring information according to the request date, request user id, request type and result status. Access Control – Several kinds of users (admin, data retrieval, product retrieval, …)

User Metrics at ECMWF

User Metrics at ECMWF : User Metrics at ECMWF

User Metrics at ECMWF : Data volumes

Introduction to Poster Session demonstrations International TIGGE Archive Centers ECMWFhttp://tigge.ecmwf.int NCARhttp://tigge.ucar.edu CMAhttp://wisportal.cma.gov.cn/tigge Poster Sessions Demonstrations When Tuesday, 15:00-16:30 (immediately following presentation) Wednesday, 16:30-18:30 Any time, please ask Baudouin, Xiaofeng or Doug Purposes Illustrate all features of the portals Register users Submit data requests Demonstrate analysis tools 3rd Thorpex Int. Science Symp., May 2009

Credits Alex de Almeida Fernandes, Baudouin Raoult, Beth Ebert, Bian Xiaofeng, Dan Swank, Dave Stepaniak, David Ian Brown, Don Middleton, Dong-Joon Kim, Doug Schuster, Glenn Rutledge, Hannah Wilcox, Jean Nicolau, Jim Fraser, Joerg Urban, Julio Pablo Reyes Fernandez, Justin Cooke, Lang Hongliang, Lewis Poulin, Li Xiang, Ma Qiang, Manuel Fuentes, Michel Grenier, Okki Lee, Richard Dare, Richard Swinbank, Shintaro Yokoi, Simon Thompson, Steve Emmerson, Steve Worley, Steven Anthony, Tan Le, Tian Hao, Tom Yoksas, Waldenio Almeida, Yang Xin, Young-Youn Park, Yves Pelletier, …... and many more