1. 11 Extending EMI middleware with DGs Peter Kacsuk, MTA SZTAKI kacsuk@sztaki.hu kacsuk@sztaki.hu http://edgi-project.eu Start date: 2010-06-01 Duration: 24 months EDGI is supported by the FP7 Capacities Programme under contract nr RI-261556

2. 2 EU FP7 projects on desktop grids: EDGeS → EDGI and DEGISCO 2 EDGI ARC, Unicore, CloudsARC, Unicore, Clouds QoS with CloudsQoS with Clouds Data intensive appsData intensive apps SG→DG direction supportSG→DG direction support EDGeS DG ↔SG integration:DG ↔SG integration: gLite → BOINC, XtremWebgLite → BOINC, XtremWeb BOINC, XtremWeb → gLiteBOINC, XtremWeb → gLite Compute intensive applicationsCompute intensive applications further d eveloped by supported by DEGISCO Disseminate and support EDGeS results world-wideDisseminate and support EDGeS results world-wide Green IT aspectsGreen IT aspects

3. 3 Infrastructure vision of EDGI EDGeS scope only for compute intensive applications for EGEE (gLite) EDGI scope for both compute and data intensive applications for EMI/EGI (gLite, ARC, Unicore) Extend Desktop Grids with Clouds for QoS

4. 4 EDGI architecture ARC gridEucalyptus/ Amazon ARC MCE atticmonitorAR 3GBridge attic monitor AR User IF Bridge IF Attic FS DG client attic Monitor UI DG Pro- ject submit inspect upload down- load submit cloud Volunteer/ Institutional Resources DG client attic Unicore grid CREAM MCE atticmonitorAR submit Unicore MCE gLite grid CREAM MCE atticmonitorAR

5. 5 5 gLite VO  DG system via EDGI services gLite WMS CE with Bridge client gLite UI EDGI Application Repository Get EXE URL Watch Get output EDGI Executor gLite BDII (with DG-CE) Info provider Report resources and performance Submit job Send job Submit job gLite LB Log events Send output gLite VOMS X509 proxy 3G Bridge BOINC1 DC-API plugin 3G Bridge BOINC2 DC-API plugin 3G Bridge XWeb XWeb plugin DG servers WS interface 3G WS client EDGI Services 1 2 3 4 0 Check EXE URL

6. 6 6 BOINC client Application DG Server BOINC Server Components BOINC client Application BOINC client Application BOINC Task DB Sche- duler Data server Job WU I/O files CE-DG Submitter Service Storage Service Job Database Job descr. gLite VO services I/O files gLiteDesktopGrid gLite-DG integration BOINC plugin DC-API master DC-API cli 3G Bridge 4 WS Interface 5 6 7 8 9 Queue Manager Download Manager gLite VO  DG system via EDGI services

7. 7 Extending DGs with Cloud resources 3G Bridge extended with Cloud Plugin The task of Cloud Plugin is to create and remove BOINC client instances in the Cloud Cloud Resource Manager realizes scheduling strategies when to create and remove BOINC client instances in the Cloud Applicable clouds: Amazon/EC2, OpenNebula, OpenStack, Eucalyptus

8. 8 Production SG->DG Infrastructure XtremWebDGsBOINCDGs XW-LAL(Local) XW-LRIEDGeS@home(Public) Uow EDGI Demo (Local) SZDGEDGeS@homeAlmeregridIbercivis(Public) Attic Services Data Center 3G-Br 3G-Br 3G-Br 3G-Br 3G-Br 3G-Br 3G-Br 3G-Br Service Grid VOs gLite  DG mCE cr1.edgi-grdi.eu ARC  DG mCE edgi-prod.nbi.dk EDGI Application Repository UoWCloudCICACloud Monitored Component PortalServiceMonitorService

9. 9 Snapshot status: gLite  DG CREAM mCE http://edgi.dei.uc.pt/EDGIMonitoring http://edgi.dei.uc.pt/EDGIMonitoring

10. 10 Snapshot status: EDGeS@home http://home.edges-grid.eu/home/ http://home.edges-grid.eu/home/

11. 11 Usage scenarios of EDGI technology 1. 1. Building infrastructure by using the EDGI services SG VOs can be extended by EDGI DG resources SG VOs can be extended by institutional DGs First create institutional DG system (e.g. Univ. of Westminster local DG) SG VOs can be extended by volunteer DGs SG VOs can be extended by EDGI DG resources and institutional/volunteer DGs 2. 2. Building infrastructure by installing the EDGI services SG VOs can be extended by institutional/volunteer DGs 3. 3. Using the infrastructure SG users can submit applications by their CLI SG users can submit applications via the EDGI portal Users of institutional DG systems can submit applications via the EDGI portal

12. 12 Application case-study via portal: Environmental Risks of Veterinary Medicines Establish whether currently available bioinformatics databases are a potential tool to predict the effects of pharmaceutical compounds on aquatic organisms Wmin Local DG with ~1600 PCs

13. 13 A CLI-based usage scenario for grid users Step 1: Select an application you want to execute on Grids Step 1: Select an application you want to execute on Grids Use any from the EDGI repository Use any from the EDGI repository Bring your own application Bring your own application Step 2: EDGI project performs the necessary preparations for you and as a result the application appears in the EDGI application repository Step 2: EDGI project performs the necessary preparations for you and as a result the application appears in the EDGI application repository application porting application porting registering the application in the AR and in several DGs registering the application in the AR and in several DGs infrastructure setup for your access (by the operators of the connecting grids) infrastructure setup for your access (by the operators of the connecting grids) Step 3: Go to our EDGI AR and collect info for your submission Step 3: Go to our EDGI AR and collect info for your submission Step 4: Prepare your input files, create and submit jdl Step 4: Prepare your input files, create and submit jdl Step 5: Query the status and download the outputs Step 5: Query the status and download the outputs 13

14. 14 Step 3: Collect information from EDGI AR (authorized vos, ces) 14

15. 15 Step 3 (cont.): Collect information from the EDGI AR (files) 15

16. 16 Step 4, 5: submit jdl and get results Create your my.jdl (as usual): Create your my.jdl (as usual): Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp", "pools.txt", OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; Submit, Status query, Output downloading can be performed by the well- known gLite commands: glite-wms-job-submit, glite-wms-job-status, glite- wms-job-logging-info, glite-wms-job-output, etc. Submit, Status query, Output downloading can be performed by the well- known gLite commands: glite-wms-job-submit, glite-wms-job-status, glite- wms-job-logging-info, glite-wms-job-output, etc. 16

17. 17 Latest development in EDGI to improve scalability: MetaJob concept 17 ARC grid gLite grid Eucalyptus/ Amazon ARC MCE atticmonitorAR CREAM MCE atticmonitorAR 3GBridge attic monitor AR User IF Bridge IF Attic FS DG client attic Monitor UI DG Pro- ject submit inspect upload down- load submit cloud Volunteer/ Institutional Resources DG client attic MetaJob as a single job Unfol ding Huge number of jobs Single job Demonstrated at EGI UF, 12 th of April, 2011 at both #10 with 10.000 jobs through gLite

18. 18 Step 4b: create and submit jdl Create your jdl: Create your jdl: Executable = "dsp"; Arguments = "-f 22 -i 22 -p 723 -n pools.txt"; InputSandbox = { "gsiftp://dev17-portal.cpc.wmin.ac.uk:2811/srv/edgi/1001/1102/dsp", "pools.txt", "_3gb-metajob-dsp-10000" }; OutputSandbox = {"cost.txt"}; SubmitTo = "cr1.edgi-grid.eu:8443/cream-pbs-edgidemo"; Submit: Submit: glite-wms-job-submit -o id edgi-metajob-10000.jdl 18 MetaJob definition as Extra input file

19. 19 MetaJob: input files and metajob definition Upload your individual input files to web server: Upload your individual input files to web server: http://somewhere.com/pools1.txt http://somewhere.com/pools1.txt … http://somewhere.com/pools10000.txt http://somewhere.com/pools10000.txt Create the description of your metajob: Create the description of your metajob: %Comment pools1.txt Arguments = "-i 0 -n pools.txt -f 22 -p 723“ Input = pools.txt=http://somewhere.com/pools1.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue…… %Comment pools10000.txt Input = pools.txt=http://somewhere.com/pools10000.txt=7b7eb86bf50c58cbf92dc12ff5adf7f4=9652 Queue 19

20. 20 MetaJob: Query the detailed status 20 Metajob handling rules Actual status of jobs Location of the mapping

21. 21 MetaJob: Download the results 21 Download result as usual: glite-wms-job-output -i id --dir outputs./outputs/cost.txt Extract it: tar zxvf cost.txt./outputs/ /cost.txt …./outputs/ /cost.txt See the mapping between your individual job definition and the jobids (which gives the name of directories storing the output files of your app) one subjob id

22. 22 Thank you for your attention… Contacts: Peter Kacsuk (kacsuk@sztaki.hu) (coordinator of EDGI, EDGeS) kacsuk@sztaki.hu Jozsef Kovacs (jozsef.kovacs@sztaki.hu) (deputy coordinator, technical leader) jozsef.kovacs@sztaki.hu Websites: www.edgi-project.eu www.edges-grid.eu Acknowledgements:  EDGI EU FP7 project (RI-261556)  EDGeS EU FP7 project (INFSO-RI-211727) 22