Catania Grid & Cloud Engine Mario Torrisi Istituto Nazionale di Fisica Nucleare – Sezione di Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA

Outline Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 2  The Catania Grid & Cloud Engine  Job Engine  Architecture  Features  Code Samples  Portlet for job management  MyJobs  mi-hostname-portlet  mi-parallel-app-portlet  References  Summary and conclusion Catania - 12 Marzo 2014

The Catania Science Gateway model Administrator Power User Basic User Grid/Cloud/Local middleware supported so far Science Gateway Science Gateway App. 1 App. 2 App. N Embedded Applications Users from different organisations having different roles and privileges Standard-based (SAGA) middleware-independent Grid Engine Standard-based (SAGA) middleware-independent Grid Engine

The Catania Grid & Cloud Engine Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 4 Catania - 12 Marzo 2014 Grid/Cloud Engine Users Tracking DB Science GW Interface SAGA/JSAGA API Job Engine Data Engine Users Track & Monit. Science GW 1 Science GW 2 Science GW 3 Grid/Cloud/Local MWs Liferay Portlets eToken Server New ModifiedNewModified New Modified

Job Engine Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 5  The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure;  It is compliant with the OGF SAGA standard;  It is optimized to be used in a Web Portal running an application server (e.g. Glassfish, Tomcat,…) based on J2EE;  It could be used also in stand-alone mode;  JSAGA is the SAGA implementation adopted. Catania - 12 Marzo 2014

A Simple API for Grid Applications (SAGA) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 6  SAGA is a family of related standards to define application programming interface (API) for common distributed computing functionalityapplication programming interfacedistributed computing  It is independent of the details of the underlying infrastructure (e.g., the middleware);  SAGA is an OGF specification:  SAGA is composed by:  SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.);  SAGA Adaptors: libraries providing access to the underlying grid infrastructure (adaptors are available for Globus, gLite, etc.);  N.B. SAGA defines a standard Catania - 12 Marzo 2014 We then need an implementation!

JSAGA Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 7  JSAGA is a Java implementation of SAGA developed at CCIN2P3;  JSAGA:  Enables uniform data and job management across different grid infrastructures/middleware;  Makes extensions easy: adaptor interfaces are designed to minimize coding effort for integrating support of new technologies/middleware;  Is OS indenpendent: most of the provided adaptors are written in full Java. Catania - 12 Marzo 2014

Job Engine Architecture Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 8 Catania - 12 Marzo 2014 USERS TRACKING DB Jobs Submission Jobs Check status/ Get output WT DCIs Worker Threads for Job Submission WT Worker Threads for Job Check Status MONITORING MODULE Jobs Queue WT USER TRACK & MONITOR

Job Engine - Features Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 9 Catania - 12 Marzo 2014

Job Engine Middleware Independent Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 10 Catania - 12 Marzo 2014 JSAGA supports gLite, Globus, ARC, UNICORE, etc. Adding new adaptors in JSAGA is a easy job

Job Engine Easiness Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 11  Allow to develop application able to submit jobs on the grid in a very short time;  A very intuitive API is exposed to the developers;  Support MPI applications;  The developer has only to submit the job:  The Job Engine periodically check the job status;  When the Job is done, the job output is automatically downloaded by the Job Engine in the local machine. Catania - 12 Marzo 2014

Job Engine Scalability (1/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 12  The Job Engine is able to manage a huge number of parallel job submissions fully exploiting the HW of the machine where it is installed;  It enqueues all the parallel requests received serving it according to the HW capabilities;  The Job Engine thread pools can be configured to optimally exploit the HW capabilities;  A burst of parallel job submissions cannot damage the Job Engine responsiveness thanks to the protection provided by the thread pool mechanism. Catania - 12 Marzo 2014

Job Engine Scalability (2/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 13 Catania - 12 Marzo 2014  The answer time is linear;  Response time depends on the HW capabilities and thread pools configuration.

Job Engine Performance Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 14  All the delays due to grid interactions are hidden to the final users:  The Job Engine provide asynchronous functions for each job management actions (submit, check status, download output, cancel);  Final users “feel” a response time equals to 0.  The Job Engine is able to submit thousands of jobs in a short time:  The Job Engine submit jobs using a configurable thread pool;  The Job Engine is able to submit jobs in less than 1 hour with 50 threads in the thread pool;  Increase the number of threads in the thread pool, can improve the Job Engine performance. Catania - 12 Marzo 2014

Job Engine Accounting Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 15  A very powerful accounting system is included in the job engine;  It is fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy;  The following values are stored in the DB for each job submitted:  Users;  Job Submission timestamp;  Job Done timestamp;  Application submitted;  Job ID;  Proxy used;  VO;  Site where the job is running (e.g., CE for EMI-gLite). Catania - 12 Marzo 2014

Job Engine Fault tollerance Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 16  Job Engine implements an advanced mechanism to guarantee job submission:  Developers can set an appropriate value of “shallow retry”;  Automatic re-submission mechanism when a job is aborted:  Hide every failure to the final users Catania - 12 Marzo 2014

Job Engine Collection, Parametric & Workflow Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 17  Currently the Job Engine natively support:  Collection jobs;  Parametric jobs;  N-1 workflows.  In the next releases we’ll add the support for DAG (Direct Acyclic Graph) Catania - 12 Marzo 2014

Code Example Job submission Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 18 Catania - 12 Marzo 2014 String wmsList[] = {"wms://wms-4.dir.garr.it:7443/glite_wms_wmproxy_server“, "wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server"/}; Resource Manager List GEJobDescription description = new GEJobDescription(); description.setExecutable("/bin/sh"); description.setArguments("hostname.sh"); description.setInputFiles("/home/mario/Documenti/hostname.sh"); description.setOutput(“Output.txt"); description.setError(“Error.txt"); Job Description JSagaJobSubmission tmpJSaga = new JSagaJobSubmission(description); tmpJSaga.useRobotProxy("etokenserver.ct.infn.it", "8082", "332576f78a4fe70a e90cd11f", "gridit", "gridit", true); tmpJSaga.setResourceManagerList(wmsList); tmpJSaga.submitJobAsync("mtorrisi", " :8162", 1, "gLite Test job“); Submit the Job

Code Example Workflow N-1(1/4) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 19 Catania - 12 Marzo 2014 JOB 1 JOB 2 JOB 4 JOB 3 … Final Job OUTPUT InfrastructureInfo infrastructures[] = new InfrastructureInfo[2]; String wmsList[] = { "wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server" }; infrastructures[0] = new InfrastructureInfo("gridit", "ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList, "etokenserver.ct.infn.it", "8082“, "332576f78a4fe70a e90cd11f", "gridit", "gridit"); String globusList[] = {"wsgram://xn03.ctsf.cdacb.in:8443/GW"}; Infrastructures[1] = new InfrastructureInfo("GARUDA","wsgram","", globusList, "etokenserver.ct.infn.it","8082","332576f78a4fe70a e90cd11f","gridit","gridit"); Define Infrastructures

Code Example Workflow N-1(2/4) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 20 Catania - 12 Marzo 2014 for (int i = 0; i < 3; i++) { GEJobDescription description = new GEJobDescription(); description.setExecutable("/bin/sh"); switch (i) { case 0: description.setArguments("hostname.sh"); description.setInputFiles("/home/diego/test_wf/hostname.sh"); break; case 1: description.setArguments("ls.sh"); description.setInputFiles("/home/diego/test_wf/ls.sh"); break; case 2: description.setArguments("pwd.sh"); description.setInputFiles("/home/diego/test_wf/pwd.sh"); break; } description.set… descriptions.add(description); } Define First Level Jobs

Code Example Workflow N-1(3/4) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 21 Catania - 12 Marzo 2014 GEJobDescription finalJobDescription = new GEJobDescription(); finalJobDescription.setExecutable("/bin/sh"); finalJobDescription.setArguments("ls.sh"); String tmp = ""; for(int i = 0; i < descriptions.size(); i++){ if(tmp.equals("")) tmp=descriptions.get(i).getOutput(); else tmp+=","+descriptions.get(i).getOutput(); } finalJobDescription.setInputFiles(tmp + ",/home/diego/test_wf/ls.sh,/home/diego/test_wf/ifconfig.sh"); finalJobDescription.setOutput("myOutput-FinalJob.txt"); finalJobDescription.setError("myError-FinalJob.txt"); Define Final Job

Code Example Workflow N-1(4/4) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 22 Catania - 12 Marzo 2014 JobCollection wf = new WorkflowN1("scardaci", "Workflow N-1", "/tmp", descriptions, finalJobDescription); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission(wf); tmpJobCollectionSubmission.submitJobCollection(infrastructures, " :8162", 1); Submit the Workflow

Grid & Cloud Engine javadoc Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 23 Catania - 12 Marzo

Job Management portlet MyJobs Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 24 Catania - 12 Marzo 2014 Active Jobs Job Status and Output Search on job description Job description

Job Management portlet mi-hostname-portlet Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 25 Catania - 12 Marzo 2014

Job Management portlet mi-parallel-app-portlet Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 26 Catania - 12 Marzo 2014

References Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 27 Catania - 12 Marzo 2014  The Catania Science Gateway Framework:  Grid & Cloud Engine Javadoc: science-gateways.it/training-materialhttp:// science-gateways.it/training-material  A Simple API for Grid Applications (SAGA):  JSAGA:

Thank you ! Catania - 24 June 2013 Webinar on the DCH-RP e-Culture Science Gateway 28 For more information on Catania Science Gateway framework please visit: