WRF4G The Weather Research Forecasting model workflow for the GRID Department of Applied Mathematics & Computer Sciences University of Cantabria, Spain GRID developers: Valvanuz Fernández, Antonio S. Cofiño Application developers: Jesús Fernández, Lluís Fita Santander Meteorology Group A multidisciplinary approach to weather & climate EGU General Assembly 2010, Wien, 2-7 May 2010

A multidisciplinary approach to weather & climate Santander Meteorology Group Numerical Weather Prediction model (regional)‏ Open source (Fortran90)‏. A world community with registered users. Application in a wide variety of studies:  Weather forecasting ( operational at NCEP )‏  Data assimilation studies  Regional climate studies  Idealized simulations  Educational applications,.... WRF Model

A multidisciplinary approach to weather & climate Santander Meteorology Group Computing and storage requirements:  Intensive use of CPU  Supports a variety of parallel programming paradigms (OpenMP, MPI, serial,...). It's a common benchmarking application for HPC.  Large amount of input, output and restart data.  Typical experiments last for days (even in parallel).  Application with a complex workflow (preprocessing, execution & postprocessing)‏ WRF Requirements metgridwrf geogrid ungrib WRF Preprocessing System WRF ARW WRFGEL namelist.input namelist.wps real WRF WORKFLOW

A multidisciplinary approach to weather & climate Santander Meteorology Group Reanalisys/Reforecasting  High number (~10 4 ) of independent simulations  High volume of output-data (>TB)  Requires scalability Regional projections for climate change  Contiguous simulations (~10), weeks of walltime each.  High volume of output data (>10TB)  Recovering system for simulation restart Weather Forecasting  QoS and optimal resources: deadline for delivering Sensitive studies for Climate and Weather  Physical schemes, initial conditions and boundary conditions uncertainties sampling  Resource demanding experiments Scientific experiments WRF

A multidisciplinary approach to weather & climate Santander Meteorology Group GOALS  Make easier for the user the process of design, execution and monitoring of the experiments with WRF.  Develop a framework that allow the user to use at the same time different computing resources in a transparent way: Local clusters (PBS, SGE, LoadLeveler,…) Grid infrastructures (gLite, Globus,…) Local resources (SSH, fork, …).  Develop a set of command line tools and a Web portal for the WRF users. WRF4G: Goals

A multidisciplinary approach to weather & climate Santander Meteorology Group CORDEX (COordinated Regional climate Downscaling EXperiment) is a framework to improve coordination of international efforts in regional climate downscaling research. CORDEX was initiated as a result of the Task Force on Regional Climate Downscaling, formed by the World Climate Research Program (WCRP). A set of target regions has been proposed and modeling groups willing to contribute must comply with simulations specifications. CORDEX will produce an ensemble of simulations sampling uncertainties related to: (i) varying Global Climate Model (GCM) simulations; (ii) varying greenhouse gas (GHG) concentration scenarios; (iii) natural climate variability; and (iv) different downscaling methods. There are currently 15 groups planning to contribute to CORDEX with WRF, 5 groups planning to contribute the African domain (key region for the AR5). All of them could benefit from WRF4G app. Role of WRF4G in CORDEX

A multidisciplinary approach to weather & climate Santander Meteorology Group CPUs 70 PB 260 sites worldwide Arquitectures: i386,x86_64 LRMS: torque, sge, lsf, bqs Shared and not shared Home Computing resources:  OS: Debian, SL/Centos 4 and 5.  Memory: 250MB to 16GB  Processor: P4 to i7 Different queues limitations: walltime, memory & disk quotas… Bandwidth in some sites is very small. EGEE: Example of big Grid infrastructure

A multidisciplinary approach to weather & climate Santander Meteorology Group GRID Challenges:  Develop an application adapted to run in different arquitectures, OS, LRMS and parallel environments.  Repository of WRF binaries adapted to those environments  Detect sites that not accomplish application requirements: Memory, Processor, queue time limitations,…  Optimizing the data transfers and replication between resources distributed geographically.  Develop a monitoring tool for experiments status.  Create a checkpointing management system that allow to restart simulations.  Failure detection and recovery. WRF4G: GRID Challenges

A multidisciplinary approach to weather & climate Santander Meteorology Group WRF4G is a port of the WRF Modeling System prepared to run in GRID environments. WRF has been splitted in 2 layers to separate the scientific experiment from the infrastructure details (wrf.input & wrf4g.conf). WRF4G supports several data transfer services (gsiftp, rsync,…) and execution systems (gLite, globus, PBS, SGE,…). It can be run in the user’s machine. No deployment required on sites. The application is deployed by a pilot job. WRF for GRID: WRF4G

A multidisciplinary approach to weather & climate Santander Meteorology Group metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW namelist.inputnamelist.wps WRF4G Workflow

A multidisciplinary approach to weather & climate Santander Meteorology Group metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW Processed locally wrf4g.conf wrf.input namelist.inputnamelist.wps WRF4G WRF4G Workflow

A multidisciplinary approach to weather & climate Santander Meteorology Group metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW WRFGEL WRF4G.sh Processed locally WRF4G_ini.sh wrf4g.conf wrf.input namelist.inputnamelist.wps wrf4g_submitter.sh Job submission abstraction layer WRF4G UI WN Input data abstraction layer WRF4G Workflow

A multidisciplinary approach to weather & climate Santander Meteorology Group metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW WRFGEL WRF4G.sh Processed locally WRF4G_ini.sh wrf4g.conf wrf.input namelist.inputnamelist.wps wrf4g_submitter.sh Job submission abstraction layer WRF4G UI WN WRF4G tgz WRF4Gbin-3.1r83.tgz wrfout_* wrfrst_* wrfinput_* wrfbdy_* WRF4G_BASEPATH WRF4G_INPUT WRF4G_APPS Input data abstraction layer geo_em.nc namelist.wps WRF4G_DOMAINS WRF4G Workflow

A multidisciplinary approach to weather & climate Santander Meteorology Group ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

A multidisciplinary approach to weather & climate Santander Meteorology Group These are sample configuration files for a WRF4G experiment ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

A multidisciplinary approach to weather & climate Santander Meteorology Group These are user scripts launched from the UI ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

A multidisciplinary approach to weather & climate Santander Meteorology Group This is what the Worker Node sees just before running WRF4G.sh ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

A multidisciplinary approach to weather & climate Santander Meteorology Group User Interface - my_experiments/exp wrf4g.conf wrf.input Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group User Interface - my_experiments/exp exp__...exp__reaN wrf4g.conf wrf.input sandbox.tgz wrf.chunk wrf.input wrf4g.conf WRF4G_ini exp__rea1 realizations Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time Three realizations split into two chunks each: exp__rea1 exp__rea2 exp__rea3 Storage Element - ${WRF4G_BASEPATH}/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfrst_t2 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

A multidisciplinary approach to weather & climate Santander Meteorology Group To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:  Data Management vcp: Provides transparent copy between any of the following protocols: gsiftp, LFC, rsync, local copies vcp -r /local/dir gridftp://server:port/remote/path vcp gsiftp://srv1:port/rmt/file gsiftp://srv2:port/other/file vcp /local/file1 rsync://server/other/local/path/ vcp gsiftp://srv1:port/rmt/file ln:link WRF4G architecture (I)‏

A multidisciplinary approach to weather & climate Santander Meteorology Group To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:  Execution Management Through the use of plugins, the user can submit jobs to a different infrastructures: Grid (globus 4.2 and glite CE), local clusters (PBS, SGE, SLURM,…), local resources. Prepared to use different execution environments: OpenMP and MPI. Strong scheduling policy based in history records and resources characteristics. Failure detection and recovery. WRF4G architecture (II)‏

A multidisciplinary approach to weather & climate Santander Meteorology Group The application is been supported in EELA-2, EGEE & NGI-ES infrastructures and GT2.4 & GT4. A release candidate (v1.0 RC1) has been launched to run realistic experiments. It consist on a virtual machine UI with all the application components. Collaborations with end-users in LA ( CETA-CIIFEN- UPS ), Europe ( CESGA-MeteoGalicia ) for feedback and Asia ( HAII, Thailand ). Currently we are testing MPI support in the GRID. Daily operational version running on the GRID for ensemble of weather forecasts. Conclusions & Current Work

A multidisciplinary approach to weather & climate Santander Meteorology Group Develop a monitoring system oriented to app status. Execution management works on Globus and gLite CE based Gridway plugins. PBS y SGE are used, but plugins will developed for better integration with Gridway. Develop an scalable replica management integrated with Gridway. (bottleneck in GRID infrastructures). Create a user portal that allow the users managing and monitoring their experiments. Collaborate with EGI and other International GRID Initiatives like EUAsiaGrid Incorporate users feedback Future Work

A multidisciplinary approach to weather & climate Santander Meteorology Group Thanks !!!