1. www.see-grid-sci.eu SEE-GRID-SCI WRF-ARW model: Grid usage The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338 Davor Davidović, Centre for Informatics and Computing Ruđer Bošković Institute Croatia

2. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 2/x Program  Introduction  Storage schema  WRF-ARW package  Data upload  Submitting model  Retrieving model data

3. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 3/x Why WRF-ARW on grid? Accessibility of “free” resources The possibility of running model on larger number of CPUs – shorter execution time The possibility of storing large amount of data (grid storage elements) ‏ The possibility of producing more accurate forecast (better resolution, shorter time interval, etc...) ‏

4. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 4/x Storage schema (1) ‏ Unified system of storing and collecting meteorological data (initial and boundary conditions, model output results, etc...) Use LFC file catalogue for storing all data WRF-ARW execution within METEO VO  Every VO has his own unique root directory within LFC with the form: /grid/VO_NAME/

5. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 5/x Storage schema (2) ‏ Before start using LFC we have to set a proper LFC server: We can check our file structure using: $ lfc-ls –l /grid/meteo.see-grid-sci.eu $ export LFC_HOST=grid02.rcub.bg.ac.yu

6. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 6/x Storage schema – example of use Root folder for WRF model data in LFC: /grid/meteo.see-grid-sci.eu/WRF-ARW Consists of 3 main sub-folders:  /bin  /input_data  /output_data

7. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 7/x Storage schema - /bin Save all executable and auxiliary files Divided in two folders  WPS – executables for pre-processing  WRF – executables for WRF-ARW core Naming convention for tar-ed executables: WRFV$VERSION_$ARCH.tar.gz WPSV$VERSION_$ARCH.tar.gz WRF_STATIC.tar.gz i WPS_STATIC.tar.gz contain all auxiliary static files These data are rarely changed ‏ (only with new versions of model)

8. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 8/x Storage shema – input_data Store all input model data (geo and grib)  /terrain – static geographical data  /raw – raw data  /precompiled – data processed with geogrid.exe  /boundary - initial and boundary conditions  /REGION – region sub-folder (user-defined) – /raw – raw data from global model servers (NCEP, ECMWF, etc...) – /precompiled – data processed with ungrib.exe All boundary condition data must be saved within a REGION

9. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 9/x Storage schma – output_data Save model output data All data are stored in REGION sub-folder defined by the user Consists of:  /REGION – region of interest (user-defined)  /sci – results of scientific mode  /oper – results of operational mode In operational mode outputs are saved by the current date Naming convention for zipped model outputs $username_WRF_ARW_YYYYMMDD_$code.tar.gz For example full path to output data: /grid/meteo.see-grid-sci.eu/WRF-ARW/output/CROATIA/oper/ /20090714/ ddavid_WRF_ARW_20090714_053009.tar.gz

10. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 10/x WRF-ARW grid workflow

11. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 11/x Before submitting model Before using scripts one must set environment variables:  WRF_HOME – full path to WRF_ARW folder  PATH – putting WRF_ARW/bin into PATH  For example : $ export WRF_HOME=/home/home_folder/WRF_ARW $ export PATH=$PATH:$WRF_HOME/bin

12. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 12/x WRF-ARW uploading data (1) ‏ There are three ways how to upload data (initial and boundary conditions):  Using standard lcg-* tools  Using DM-Web web application  Using wrf-upload-data script Example of copying data from UI node to specified grid SE ‏ $ lcg-cr --vo $VO –d $SE $LOCAL_FILE –l lfn:/LFC_LOCATION/FILE_NAME

13. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 13/x WRF-ARW uploading data (2) ‏ The second way by using DM-Web application  Demo example web-interfaceweb-interface By using scripts  wrf-upload-data (most appropriate for downloading data for operational forecast) ‏ $ wrf-upload-data –h $ wrf-upload-data –l limit –d destination For uploading data for scientific mode use lcg-* tools

14. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 14/x WRF-ARW submit Submit model on the WMS (grid) by using this script: $ wrf-submit [options] Simply to use Automatically generates *.jdl script and submit job on the grid Complexity of gLite middleware is hidden from end user $ wrf-submit -h

15. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 15/x WRF-ARW submit Options for starting model:  -m – type of the model run (options: o – operational, s – scientific) ‏  -r – name of the region  -d – SE for storing output model outputs  -t – set the Vtable (default values: GFS for operative, ECMWF for scientific) ‏  -p – number of processors  -i – name of the input file  -o – name of the output file (obligatory for scientific mode) ‏

16. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 16/x WRF-ARW operational mode How to start in operational mode  Script automatically updates start and end dates in namelist files  Start model with executing command:  Generates file with grid job id (extension *.job) given by WMS  namelist.input and namelist.wps must be located in the folder from which we start wrf-submit command $ wrf-submit –m o –p $NUM_PROC $ wrf-submit –m o –p 8 –o $USERNAME

17. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 17/x WRF-ARW scientific mode Starting scientific mode  namelist.input and namelist.wps must be located in folder from which we run wrf-submit command  The user must manually set parameters in namelist files (start, end dates, period of the simulation, etc. )  The simplies way to start a model:  example:  Also generates files with saved grid job id $ wrf-submit –m s –i $INPUT_FILE $ wrf-submit –m s –i test_sci –p 8 –o $USERNAME

18. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 18/x Checking job state Name format of generated jobid file: username_YYYY-MM-DD_run_run-id.job Jobid files in necessary for checking your job status Check job status by using gLite tools: Check job status by using script: $ glite-wms-job-status –i JOBID_FILE.job $ wrf-job-status –i JOBID_FILE.job

19. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 19/x WRF-ARW retrieving outputs After the run is finish all output results are packed and stored on (user defined or default) grid SE within LFC catalogue: /WRF-ARW/output_data/REGIJA/sci/ ili /WRF-ARW/output_data/REGIJA/oper/YYYYMMDD/ Script for downloading output files: Downloading is possible on UI node or some other user-defined server $ wrf-get-data –i JOBID_FILE.job

20. SEE-GRID-SCI Kick-off meeting, Athens, May 22-23, 2008 20/x Thank you for Your attention! Questions?