1. www.consorzio-cometa.it FESR Consorzio COMETA - Progetto PI2S2 Porting MHD codes on the GRID infrastructure of COMETA Germano Sacco & Salvatore Orlando Consorzio COMETA/INAF-OAPA sacco@astropa.inaf.it PRIMO TUTORIAL GRID PER L’UNIVERSITA’ DI PALERMO Palermo, 10-12 Dicembre 2007

2. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 Summary Introduction about MHD codes and FLASH FLASH porting on COMETA: main problems FLASH porting on COMETA: solutions The PLUTO code and the first test on COMETA

3. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 3 MHD equations  : density u: velocity P : pressure E : Energy B: magnetic field Other terms: gravitational field thermal conduction radiative losess viscosity magnetic resistivity nuclear burning Also relativistic formulation (RMHD)

4. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 4 Some astrophysical problems Supernova Remnants Accretion phenomena in young stars Protostellar jets Coronal Mass ejections in the solar atmosphere (see, for instance, Bonito et al., Miceli et al., Orlando et al., Pagano et al., Sacco et al. presentation and posters at the GRID open days) Supernova Remnant Cas A

5. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 5 The FLASH Code Developed at the ASC FLASH Center (University of Chicago) (see Fryxell et al.2000) Written in Fortran 90 and C Run on parallel computers (MPICH libraries) Modular structure PPM algorithm for HD eq. solving, Powel algorithm for MHD eq. solving Adaptive Mesh refinement (Paramesh 2.0, MacNeice et al. 2000) HDF5 libraries for I/O (dynamical link) production of Checkpoint files during the run

6. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 6 FLASH porting on COMETA: main problems Compatibility between FLASH and the Hardware/Software architecture of the COMETA worker nodes Dynamical link with the HDF5 libraries Job monitoring, checkpoint files and big output files Jobs longer than 12h (execution time), requiring > 5000h (CPU time summed on all the processors used) Jobs using many CPUs (> 32) FLASH performance on the GRID (to be optimized)

7. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 7 FLASH/COMETA compatibility FLASH must be compiled on the site UI (e.g. unipa-ui- 01.pa.pi2s2.it) in order to build the package with the MPI libriaries MPI2 Libraries at /opt/share/mvapich2/ PGI Compiler 7.0-6 64-bit target on x86-64 Linux (Cc, CC, F77, F90, F95)

8. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 8 Dynamical link: HDF5 library HDF5 libraries statically installed on each GRID WN (LD_LIBRARY_PATH=/opt/exp_soft/hdf5/5-1.6.5/lib) Dynamic installation doesn't work HDF5_PATH = /opt/exp_soft/hdf5/5-1.6.5 LIB_HDF5 = -L $(HDF5_PATH)/lib -lhdf5 -lz You must change the set for HDF5_PATH and LIB_HDF5 in your makefile:

9. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 9 Job Monitoring 1.Look at the working directory every 300 seconds (more or less frequently if you want) 2.Copy log files on your own grid directory in a storage element 3.If there are new checkpoint files, copy them on your directory in a storage element 4. At the end of the job, create a tar file with all the output files and copy them on your directory in a storage element The watchdog script

10. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 10 Long & many cores Job voms-proxy-init --voms cometa:/cometa/hpc –valid 504:00 You must be included among the cometa HPC users: You may submit the job on HPC queue on the unipa site: unipa-ce-01.pa.pi2s2.it:2119/jobmanager-lcglsf-hpc Requirements: Execution Time > 1 week Number of processors > 32

11. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 11 mpi.pre.sh echo "--------- mpi.pre.sh ---------" export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/exp_soft/hdf5/5-1.6.5/lib/ echo "Starting at: "$(date +%y%m%d%H%M%S) HOSTNAME=$(hostname -f) USER=$(whoami) ARG1=$1 LOCALDIR=$(pwd) # Start the watchdog chmod +x watchdog.sh /bin/bash watchdog.sh > watchdog.out &

12. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 12 mpi.post.sh echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" # Close the watchdog # ---------------------------------------------- # stop and wait the wdog rm -f watchdog.ctrl while [ ! -e watchdog.done ] do sleep 1 echo "Waiting for watchdog: "$(date +%y%m%d%H%M%S) done echo "Watchdog closed" echo "done" echo "done" 1>&2 echo "Ending at: "$(date +%y%m%d%H%M%S)

13. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 13 mpi.post.sh # Transfer the FLASH data to SE WD_CATALOGPATH=/grid/cometa/orlando/FLASH_OUT/ export LFC_HOME=$WD_CATALOGPATH export WD_CD=$(basename ${PWD} | awk '{ l=length($1); print substr($1,l-21,22) }') lfc-mkdir $WD_CD # create and gzip the archive files tar -cvf flash_chk.tar *chk* tar -cvf flash_plt.tar *plt* gzip flash_chk.tar gzip flash_plt.tar # transfer the gzipped archive files to SE lcg-cr --vo cometa file:$PWD/flash_chk.tar.gz -l lfn:$WD_CD/flash_chk.tar.gz -d unipa-se-01.pa.pi2s2.it -P generated/hpc/fileff`uuidgen` lcg-cr --vo cometa file:$PWD/flash_plt.tar.gz -l lfn:$WD_CD/flash_plt.tar.gz -d unipa- se-01.pa.pi2s2.it -P generated/hpc/fileff`uuidgen

14. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 14 flash.jdl Type = "Job"; JobType = "MVAPICH2"; NodeNumber = 16; Executable = "flash2"; StdOutput = "mpi.out"; StdError = "mpi.err"; InputSandbox = {"watchdog.sh", "mpi.pre.sh", "mpi.post.sh", "flash.par", "summers_den_1e0.rates", "flash2"}; OutputSandbox = {"mpi.err", "mpi.out", "watchdog.out", "flash_oph.log", "amr_log"}; Requirements = (other.GlueCEUniqueId == "unipa-ce- 01.pa.pi2s2.it:2119/jobmanager-lcglsf-hpc"); MyProxyServer = "grid 001.ct.infn.it"; RetryCount = 3;

15. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 15 FLASH performance on COMETA MPI libraries compilation on COMETA GRID has to be optimized FLASH performance on COMETA are worse than on OAPA/PAX

16. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 16 PLUTO astrophysical code Developed at the Physical Department of the University of Torino and at INAF-Turin Observatory by Andrea Mignone with the contribution of T. Matsakos, O. Tesileanu, G. Bodo and C. Zanni (Mignone et al. 2007) Written in C Solve MHD and RMHD equations in different geometries Use different equations solving algorithms Run on parallel machines (MPI libraries) modular structure No dynamical links with external libraries

17. Palermo, Grid Tutorial per l'Universita' di Palermo, 10.12.2007 17 First test of PLUTO on COMETA 1. PLUTO compiles correctly on the unipa-ui-01.pa.pi2s2.it User Interface 2. PLUTO runs correctly on the GRID, when all the processors belong to the same Worker Node (max 4 CPUs) 3. PLUTO seems to run correctly on the GRID, also using more than 4 CPUs but output files are not complete. PLUTO uses a parallel I/O