1. www.epikh.eu The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) GISELA Additional Services Diego Scardaci (diego.scardaci@ct.infn.it)diego.scardaci@ct.infn.it INFN Dept. of Catania Joint CHAIN/GISELAEPIKH Application Porting School Valparaiso (Chile), 29.11-2010-09.12.2010

2. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 2 Outline The Secure Storage Service for the gLite Middleware Watchdog LCG-REC OurGrid

3. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 3 Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way. The service is composed by the following components: Command Line Applications: commands integrated in the gLite User Interface to encrypt/upload and decrypt/ download files. Application Program Interface: allows the developer to write programs able to manage confidential data. Keystore: a new grid element used to store and retrieve the users’ keys. Secure Storage

4. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 4 Using Secure Storage Secure Storage is a GISELA infrastructure service; The Secure Storage libraries are been installed in the GISELA Central UIs and in the GISELA sites; A central Keystore has just been installed in the machine securestorage-01.ct.infn.it; To use Secure Storage: Read the documentation in the wiki site: http://grid.ct.infn.it/twiki/bin/view/EELA2/SecureStorage; http://grid.ct.infn.it/twiki/bin/view/EELA2/SecureStorage See the code samples; Contact: Diego Scardaci (mail: diego.scardaci@ct.infn.it)diego.scardaci@ct.infn.it

5. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 5 Set the following environment variables in the UI or in the WN: SS_GRID_KEYSTORE_HOST= :25406 SS_GRID_KEYSTORE_DN= In the main script of your application, set the following environment variables to be able to use Secure Storage in a WN: PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorage- client/bin/:${PATH} LD_LIBRARY_PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestora ge-client/lib/:${LD_LIBRARY_PATH} [ Type = "Job"; JobType = "Normal"; Executable = "/bin/sh"; Arguments = "MySSApplication.sh"; StdOutput = "MySSApplication.out"; StdError = "MySSApplication.err"; InputSandbox = {“MySSApplication.sh”,…}; OutputSandbox = {"MySSApplication.err"," MySSApplication.out"}; ] MY JOB #!/bin/sh … export SS_GRID_KEYSTORE_HOST= :25406 export SS_GRID_KEYSTORE_DN= export LCG_CATALOG_TYPE=lfc export LFC_HOST=lfc.eela.ufrj.br export LCG_RFIO_TYPE=dpm export PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorage- client/bin/:${PATH} export LD_LIBRARY_PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/secure storage-client/lib/:${LD_LIBRARY_PATH} #run application Using Secure Storage

6. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 6 An example of Secure Storage Job #!/bin/sh export SS_GRID_KEYSTORE_HOST=securestorage-01.ct.infn.it:25406 export SS_GRID_KEYSTORE_DN="/C=IT/O=INFN/OU=Host/L=Catania/CN=securestorage-01.ct.infn.it“ export LCG_CATALOG_TYPE=lfc export LFC_HOST=lfc.eela.ufrj.br export LCG_RFIO_TYPE=dpm export PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorage-client/bin/:${PATH} export LD_LIBRARY_PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/securestorage- client/lib/:${LD_LIBRARY_PATH} echo "SECURE STORAGE TEST" > text_file_5.txt lcg-scr --vo prod.vo.eu-eela.eu -d lnx097.eela.if.ufrj.br --vo_permission /C=IT/O=INFN/OU=Personal\ Certificate/L=Catania/CN=Diego\ Scardaci -l lfn:/grid/prod.vo.eu-eela.eu/text_file_6.enc text_file_5.txt lcg-scp --vo prod.vo.eu-eela.eu lfn:/grid/prod.vo.eu-eela.eu/text_file_6.enc file:$PWD/text_file_copy_dec.txt echo "That's all folks!“ exit $? MY JOB MY SCRIPT [ Type = "Job"; JobType = "Normal"; Executable = "/bin/sh"; Arguments = "./securestorage_test.sh"; StdOutput = "out-securestorage_test.out"; StdError = "err-securestorage_test.err"; InputSandbox = {"./securestorage_test.sh"}; OutputSandbox = {"text_file_copy_dec.txt","err- securestorage_test.err","out-securestorage_test.out"}; ]

7. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 7 #include "securestorage.h“ … int main (int argc, char *argv[]) { int fd; int read_byte=0; unsigned char *buffer; FILE *fdout; if (argc != 3) { fprintf(stderr,"Usage: %s \n",argv[0]); exit(1); } if ( (fdout = fopen(argv[2], "wb")) == NULL ) { perror("fopen"); exit(1); } if ( (fd=securestorage_open(argv[1], O_RDONLY, 0644)) < 0 ) { /*manage the error*/ } buffer = calloc(1, BUFLEN+1); do { if ((read_byte=securestorage_read(fd,buffer,BUFLEN))<0){ /*manage the error*/} if (read_byte>0) { if (fwrite(buffer, 1, read_byte, fdout) < 0) {/*manage the error*/} } } while (read_byte>0); if ( securestorage_close (fd) < 0 ) {/*manage the error*/ } fclose(fdout); free(buffer); exit(0); } An example of Secure Storage API usage inside a C program

8. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 8 Why –Especially long term jobs require to be monitored and controlled during their execution. How –Perform job control and monitoring using grid services in the less invasive way. Observations –Almost all jobs submitted on the grid are piloted by shell scripts  Shell scripting allow to get precious info in case of faults  Shell scripting can pilot more complex batch execution –Both AMGA and SE+LFC can be used as the simplest IS on the grid.  lfc-* and lcg-* tools already available for file creation and retrieve  The latency of CLI tools for the storage is very low compared to long term jobs Requirements: –Monitor job execution watching snapshot of files produced by the job execution  File snapshot will be reported on LFC+SE or AMGA servers –It would be useful to configure the monitoring tool accordingly to the user needs  Few shell environment variables can be used to configure the watchdog tool –Control the job execution accessing directly on the WN  It is possible to send commands on the WN Watchdog

9. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 9 Watchdog The Watchdog is a shell script to be included in the JDL main script. –Some watchdog features:  It starts in background before to run the long term job  The watchdog runs as long as the main job  The main script can control, stop and wait until the watchdog has finished  Easily and highly configurable and customizable  The watchdog does not compromise the CPU power of the WN Amga LFC+SE Commands or Scripts JDL WN File snapshots WN UI

10. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 10 Using the Watchdog Download Watchdog from GISELA forge: https://forge.eu- eela.eu/frs/?group_id=103https://forge.eu- eela.eu/frs/?group_id=103 Read the documentation in the wiki site: http://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog http://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog See the code samples distributed inside the packages: script.jdl script.sh Add Watchdog capabilities to your application! Contact: Riccardo Bruno (mail: riccardo.bruno@ct.infn.it)

11. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 11 Using the Watchdog Configure the Watchdog setting the watchdog.conf file (see the wiki); Applications using Watchdog MUST include the files watchdog.sh, watchdog.ctrl, watchdog.conf. [ Type = "Job"; JobType = "Normal"; Executable = "/bin/bash"; StdOutput = "file.out"; StdError = "file.err"; InputSandbox = {"watchdog.sh","watchdog.ctrl","watchdog.co nf",“MyScript.sh"}; OutputSandbox = {“MyApplication.out","MyApplication.err","wat chdog.log”}; Arguments = "script.sh"; ] MY JOB #!/bin/sh … # prepare and start the watchdog PATH=${VO_PROD_VO_EU_EELA_EU_SW_DIR}/ :${PATH} chmod +x watchdog.sh chmod +x watchdog.ctrl chmod +x watchdog.conf./watchdog.ctrl start... #run application … #stop and wait the watchdog completes./watchdog.ctrl stop MyScript.sh

12. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 12 LCG-REC https://grid.ct.infn.it/twiki/bin/view/GILDA/Lcg-rec-tools The new tools have the following prefix: 'lcg-rec-*' to enphatize the recursive interaction with the calalog/SE. These tools are described briefly below: lcg-rec-cr - Exactly like the corresponding lcg-cr command, this tool copy and register a whole directory structure from a local filesystem in the UI to the file catalog. The same file hierarchy will be kept in the catalog and the related file contents will be stored on a storage element. lcg-rec-cp - Exactly like the corresponding lcg-cp command, this tool copy from catalog a directory structure downloading the related file content from the storage elements. The same file hierarchy will be kept in the local filesystem. lcg-rec-del - Exactly like the corresponding lcg-del command, this tool removes a entire directory structure from the catalog and removes each replica of related file from the storage elements. lcg-rec-rep - Exaclty like the corresponding lcg-rep command, this tool adds recursively a replica to all files of a directory.

13. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 13 Production opportunistic infrastructure powered by OurGrid OurGrid

14. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 14 OurGrid Production gateway allowing the service and the opportunistic infrastructure to interoperate

15. Valparaiso, Joint CHAIN/GISELA/EPIKH Grid School for Application Porting, 01-12-2010 15 Thank you very much!