The EDGI project receives Community research funding 1 Desktop Grid Infrastructure and User Support Services for EGI/NGI User Communities Tamas Kiss – University of Westminster, UK Zoltan Balaton – SZTAKI, Hungary

Presentation title: EGI TF Author: Tamas Kiss 2 Desktop Grid related services and resources offered for EGI/NGI users A production infrastructure bridging from g-Lite (ARC, UNICORE) resources to BOINC and XtremWeb based DGs DG worker nodes (over 100,000 non-dedicated workers) Support for infrastructure providers connecting SG and DG resources to the existing infrastructure Setting up SG/DG infrastructures based on EDGI technology Support for application developers porting applications to SG/DG platform Support for users Training Running the application on SG/DG resources

Presentation title: EGI TF Author: Tamas Kiss 3 The EDGeS Integrated SG–DG Infrastructure BOINC-based DGs XtremWeb/XWHEP-based DGs local XtremWeb DG IN2P3 993 PCs new public XtremWeb DG 70 PCs EGEE VO public XtremWeb DG IN2P3 21 PCs public XtremWeb DG AlmereGrid 50 PCs local BOINC DG Westminster PCs local BOINC DG Correlation Systems public BOINC DG AlmereGrid PCs public BOINC DG Ibercivis PCs new public BOINC DG PCs public BOINC DG SZDG PCs public BOINC DG Extremadura PCs (of avail.)

Presentation title: EGI TF Author: Tamas Kiss 4 Current Production DG->SG Infrastructure voms lbwmsbdii BOINC- EGEE bridge (ui1) XtremWeb-EGEE bridge (xw2) BOINC-based DGs UoW, Correlation Systems local DGs SZDG, Almere, Ibercivis public DGs XtremWeb- based DGs IN2P3, Almere local DGs IN2P3 public DGs SZTAKI 16 cpus AR UI/ Bridge Core Service Resou rce CIEMAT 20 cpus ce CNRS 1800 cpus ce lfc se mypro xy Sinica 6 ce LIP 1080 ce BIFI 36 ce IPB 672 ce UFCG 6 ce KFKI 410 ce grid.edges-grid.eu

Presentation title: EGI TF Author: Tamas Kiss 5 Current production SG ->DG Infrastructure voms lbwmsbdii gLite based service grid EGEE-DG bridge (ce1) BOINC-based DGs UoW local DGs SZDGr, Almere, Ibercivis public DGs XtremWeb- based DGs IN2P3 local DG Almere public DGs AR Bridge Core Service Resou rce lfc bridge mypro xy bridge VO1VO2

Presentation title: EGI TF Author: Tamas Kiss 6 WN1 P-GRADE Portal Interface to SG->DG Infrastructure WS Interface Core 3G Bridge DG Server DG server machine CE+ Bridge Client AR EDGeS Bridge Services CE AR Cache EDGeS portal VOCE WMS VOCE CE Desktop 1 Desktop 2 Desktop N SEEGRID WMS WN2

Presentation title: EGI TF Author: Tamas Kiss 7 The EDGI Infrastructure – the future The EDGI Infrastructure – the future Extending EDGeS with Unicore, ARC and Cloud support XWHEP based public DGs local DGs Cloud 1 (Eucalyptus) Cloud 2 (Opennebula) BOINC based public DGs local DGs DG VO VO1 VO2 ARC based SGs VO1 VO2 Unicore based SGs VO1 VO2 DGs of the Internstional Desktop Grid Federation: SZDG, Ibercivis, AlmereGrid, IN2P3 Grid, Extremadura,... gLite SGs EDGeS

Presentation title: EGI TF Author: Tamas Kiss version: 1 8 How the jobs are bridged from SG to DG? DG side EGEE side LCG CE with EDGeS bridge JobManager DG server EDGeS Application Repository EDGeS GRAM JobManager EDGeS Submit script EDGeS AR client EDGeS Cancel script EDGeS Status script poll() cancel_in_batch_syste m() submit_to_batch_syste m() GRAM job ID gLite UI BOINC DB 3G Bridge queue manager HTTP Job HTTP Storage Job tmp storage EDGeS WSClient gLite WMS end user DC-API- SINGLE plugin 3G Bridge WSSubmitter DG admin deployed applications AR admin EDGeS AR client

Presentation title: EGI TF Author: Tamas Kiss 9 What can be bridged? Let there be a validated version of an application in the AR with executables for EGEE and different DG systems (and on DGs for different platforms)Let there be a validated version of an application in the AR with executables for EGEE and different DG systems (and on DGs for different platforms) This application (the client part) is deployed on a DG that is connected to the bridge and this DG is registered in the AR as supporting the applicationThis application (the client part) is deployed on a DG that is connected to the bridge and this DG is registered in the AR as supporting the application An EGEE VO is also registered in the AR as an allowed source of jobs for this applicationAn EGEE VO is also registered in the AR as an allowed source of jobs for this application

Presentation title: EGI TF Author: Tamas Kiss 10 How does bridging work? When a job is submitted to an EDGeS bridge CE it checks the following:When a job is submitted to an EDGeS bridge CE it checks the following: –Executable matches the one in the AR for the source VO by MD5 hash 1.The source VO must be allowed 2.The application executable must be allowed –The target DG is registered as supporting the application (the DG version is deployed there) If the above are true the job is bridged if false then the job is rejectedIf the above are true the job is bridged if false then the job is rejected

Presentation title: EGI TF Author: Tamas Kiss 11 Role of the Application Repository In SGs security is based onIn SGs security is based on –X509 public key infrastructure to authenticate users, resources and services and VOMS to authorise them –user separation on resources and administered resources In DGs security is based onIn DGs security is based on –thoroughly tested applications on all platforms and with all input data scenarios before promoting them to production status –code signing with a key stored on a network-isolated machine to prevent malicious executable distribution (applications must be deployed by hand) –result validation and redundant computing to probabilistically detect result and credit falsification –(see for summary) The AR is helping in converting between security modelsThe AR is helping in converting between security models

Presentation title: EGI TF Author: Tamas Kiss 12 What needs to be set up? On the DG side:On the DG side: –3g-bridge queue manager –3g-bridge wssubmitter service –To get applications from the AR to be installed locally and to register installed applications: gemlcacli and gridftp clients or a web browser On the EGEE side:On the EGEE side: –An lcg-CE with edges-BRIDGE –Connecting the EGEE CE to the wssubmitter(s) –To get applications from the AR to be submitted to SG gemlcacli and gridftp clients or a web browser

Presentation title: EGI TF Author: Tamas Kiss 13 Supporting application developers and end-users The Application Support Service Aims and objectives: –facilitate the porting of applications to the combined SG/DG infrastructure by targeting user communities already associated with EGEE or DGs new user communities Activities: –develop a generic methodology for application porting –identify user communities that require the power of the SG/DG infrastructure –provide a service in order to aid the migration to and running of applications on the SG/DG infrastructure

Presentation title: EGI TF Author: Tamas Kiss EADM – EDGeS Application Development Methodology 1. Analysis of current application 2. Requirements analysis 3. Systems design 4. Detailed design5. Implementation6. Testing 7. Validation 8. Deployment 9. User support, maintenance & feedback

Presentation title: EGI TF Author: Tamas Kiss 15 SG/DG Applications on Production infrastructure (ported by EDGeS) communitynumber of applications academicindustry Bioscience 761 Healthcare 211 Physics 77 Audio and video processing 321 Business 11 Applied mathematics 11 Engineering 11 Total 22193

Presentation title: EGI TF Author: Tamas Kiss 16 Which applications are suitable for SG/DG infrastructures? Applications should run on both SGs and DGs –SGs support a much wider scale of applications than DGs –We should assure that the application runs on DGs Requirements towards an SG application to be executable on DGs: –Parallelization: Only master/worker or parameter sweep parallelisation –No MPI or internal communication between worker nodes –Nodes can only use the results of other nodes through the server –Data handling: small or medium-sized (max. 100 MB per worker) inputs and outputs No shared data storage No confidential data (sent down to potentially un-trusted worker)

Presentation title: EGI TF Author: Tamas Kiss 17 To achieve good performance –the execution time of individual jobs should be Minimum: –over 10 minutes (otherwise the overhead caused by the DG will reduce the performance) Maximum: –less than minutes (if more, application level checkpointing is required to avoid loss of computation caused by user interventions) –the execution of individual jobs should take around the same amount of time (better scheduling, less load on the server) Operating systems –Depends on the DGs where the application will run windows version may be required to utilise larger number of resources Which SG applications are suitable for EDGI?

Presentation title: EGI TF Author: Tamas Kiss 18 What is an SG/DG application? An SG/DG application is capable using both Service (EGI) and Desktop Grid resources SG/DG application scenarios: 1.Application runs on DG and uses SG resources via the DG to SG bridge 2.Application submitted to SG and uses DG resources via the SG to DG bridge 3.Application uses both SG and DG resources via an external scheduling and job submission system (not through the EDGeS bridges)

Presentation title: EGI TF Author: Tamas Kiss 19 Scenario 1 – DG to SG via bridge DG user Desktop Grid EGEE EDGeS VO WMS and other EGEE services DG->EGEE bridge User entry point is DG – using SG is completely transparent from user’s point of view

Presentation title: EGI TF Author: Tamas Kiss 20 Scenario 2 – SG to DG via bridge SG (EGEE) user (using EGEE UI machine or portal) Desktop Grid 1 EGEE VO WMS and other EGEE services −User entry point is SG −using DG is transparent from user’s point of view Desktop Grid n Edges Services DG CE + EDGeS AR

Presentation title: EGI TF Author: Tamas Kiss 21 Porting to EDGI: 1.Develop DG version of application 2.Test DG version and write test report Use the EDGI Development DG Use your own development DG Test report template downloadable from EDGeS website 3.Have your application validated Validation is done by EDGI Validation Team Outcome: validation document Validation includes tests via the SG->DG test bridge 4.Publish your application in EDGI Application Repository DG Admins can download your application from EDGeS AR if they are ready to support it DG Bridge CE should be connected to your EGEE VO for execution Scenario 2 – SG to DG via bridge

Presentation title: EGI TF Author: Tamas Kiss 22 Application examples – Scenario 2 EDGeS application examples: –EMMIL E-marketplace application – run on SEE-Grid already –VisIVO - Visualisation Interface to the Virtual Observatory – EGEE application by Astrophysical Observatory of Catania, ported to DG by UoW –3D video rendering – application ported to EGEE (however, DG version already existed) –ISDEP - Integrator of Stochastic Differential Equations in Plasmas - EGEE application ported by BIFI

Presentation title: EGI TF Author: Tamas Kiss Application Example 1 VisIVO - Visualisation Interface to the Virtual Observatory What is VisIVO Server? –a suite of software tools for creating customized views of 3D renderings from astrophysical data tables –supports high-performance, multidimensional visualization Application porting –Desktop Grid version is deployed and running on UoW Local DG –Application is validated and published in EDGeS Production Application Repository –Application runs from EGEE to DG (UoW Local DG and Data distribution –Medium sized input files (up to 100 Mbytes) are currently feasible –Division of input file (potentially GBytes) and better data distribution using ADICS will be investigated in EDGI

Presentation title: EGI TF Author: Tamas Kiss 24 Scenario 3 – SG/DG resources but not through EDGeS bridges Using external job submission and scheduling system to submit jobs to both SG and DG resources –WS P-GRADE portal supports this scenario by default E.g.: –CancerGrid application uses EGEE and DG resources via the WS P-GRADE portal (but not through the bridges - see presentation on Friday afternoon) –Could be justified with specific user requirements E.g.: –WISDOM project uses only pull jobs on EGEE that are unsuitable to be bridged to DG –Both solutions use EDGeS technology: 3GBridge

Presentation title: EGI TF Author: Tamas Kiss 25 Scenario 3 – e.g. DG submitter for WISDOM WISDOM: Meta middleware to submit pull (pilot) jobs to EGEE The DG submitter: –Submit push (direct) jobs to the DG when EGEE resources are overloaded WISDOMJobManager 3G Bridge WSSubmitter 3G Bridge Bridge UoWBOINC EDGeSSubmitter Push jobs WISDOMTaskManager WISDOM Desktop Grid EGEE Submitter Pull jobs EGEE EGEE

Presentation title: EGI TF Author: Tamas Kiss 26 Thank you for your attention … Any questions?