1. The Catania Science Gateway Framework
The Catania Grid Engine and some implementations of the framework Diego Scardaci INFN
The Catania Science Gateway Framework

2. Outline The Catania Grid & Cloud Engine
Job Engine
Code Sample
MyWorkspace & MyJobs
Data Engine
Implementations of the framework
Summary and conclusions

3. The Catania Science Gateway model
Embedded Applications
Administrator
Power User
Basic User
App. 1
App. 2
App. N
Gateway
Science
Standard-based (SAGA) middleware-independent
Grid Engine
Users from different organisations having different roles and privileges
Grid/Cloud/Local middleware supported so far

4. The Catania Grid & Cloud Engine
Liferay Portlets
Grid/Cloud
Engine
Science
GW 1
Science
GW 2
Science
GW 3
eToken
Server
Science GW Interface
Data Engine
Job Engine
Users Track &
Monit.
Users
Tracking DB
SAGA/JSAGA API
Grid/Cloud/Local MWs
Modified
New
New
Modified
Modified
New

5. Job Engine
The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure;
It is compliant with the OGF SAGA standard;
It is optimized to be used in a Web Portal running an application server (e.g. Glassfish, Tomcat,…) based on J2EE;
It could be used also in stand-alone mode;
JSAGA is the SAGA implementation adopted.

6. A Simple API for Grid Applications (SAGA)
SAGA is an API that provides the basic functionality required to build distributed applications, tools and frameworks;
It is independent of the details of the underlying infrastructure (e.g., the middleware);
SAGA is an OGF specification:
Several Implementations are available:
A C++ and a Java implementation developed at the Louisiana State University / CCT and Vrije Universiteit Amsterdam (
A Java implementation developed at CCIN2P3 (
A Python implementation based on those above.

7. A Simple API for Grid Applications (SAGA)
SAGA is composed by:
SAGA Core Libraries: containing the SAGA base system, the runtime and the API packages (file management, job management, etc.);
SAGA Adaptors: libraries providing access to the underlying grid infrastructure (adaptors are available for Globus, gLite, etc.);
SAGA defines a standard
We then need an
implementation!

8. JSAGA JSAGA is a Java implementation of SAGA developed at CCIN2P3;
Enables uniform data and job management across different grid infrastructures/middleware;
Makes extensions easy: adaptor interfaces are designed to minimize coding effort for integrating support of new technologies/middleware;
Is OS indenpendent: most of the provided adaptors are written in full Java and they are tested both on Windows and Linux.

9. Job Engine - Requirements
The Job Engine has been designed with the following requirements in mind:
Feature
Description
Status
Middleware Independent
Capacity to submit job towards resources running different middleware
DONE
Easiness
Create code to run applications on the grid in a very short time
Scalability
Manage a huge number of parallel job submissions fully exploiting the HW of the machine where the Job Engine is installed
Performance
Have a good response time
Accounting
Register every grid operation performed by the users
Fault Tolerance
Hide middleware failure to the final users
Workflow
Providing a way to easily create workflow
PARTIALLY DONE

10. Job Engine - Architecture
DCIs
Jobs
Check status/
Get output
Worker Threads for
Job Check Status
MONITORING MODULE
USERS
TRACKING DB WT WT WT WT WT WT WT
Jobs
Submission WT WT WT
Jobs Queue
Worker Threads for Job Submission

11. Job Engine Middleware Independent
JSAGA supports gLite, Globus,
ARC, UNICORE, etc.
Adding new adaptors in JSAGA
is a easy job

12. Job Engine Easiness
Allow to develop application able to submit jobs on the grid in a very short time;
A very intuitive API is exposed to the developers;
Support MPI applications;
The developer has only to submit the job:
The Job Engine periodically check the job status;
When the Job is done, the job output is automatically downloaded by the Job Engine in the local machine.

13. Job Engine Scalability (1/2)
The Job Engine is able to manage a huge number of parallel job submissions fully exploiting the HW of the machine where it is installed;
It enqueues all the parallel requests received serving it according to the HW capabilities;
The Job Engine thread pools can be configured to optimally exploit the HW capabilities;
A burst of parallel job submissions cannot damage the Job Engine responsiveness thanks to the protection provided by the thread pool mechanism.

14. Job Engine Scalability (2/2)
The answer time is linear;
Response time depends on the HW capabilities and thread pools configuration.

15. Job Engine Performance
All the delays due to grid interactions are hidden to the final users:
The Job Engine provide asynchronous functions for each job management actions (submit, check status, download output, cancel);
Final users “feel” a response time equals to 0.
The Job Engine is able to submit thousands of jobs in a short time:
The Job Engine submit jobs using a configurable thread pool;
The Job Engine is able to submit jobs in less than 1 hour with 50 threads in the thread pool;
We could improve this measurement increasing the number of threads in the thread pool.

16. Job Engine Accounting
A very powerful accounting system is included in the job engine;
It is fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy;
The following values are stored in the DB for each job submitted:
Users;
Job Submission timestamp;
Job Done timestamp;
Application submitted;
Job ID;
Proxy used;
VO;
Site where the job is running (e.g., CE for EMI-gLite).

17. Job Engine Fault Tolerance
Job Engine implements an advanced mechanism to guarantee job submission:
Developers can set an appropriate value of “shallow retry”;
Automatic re-submission mechanism when a job is aborted:
Hide every failure to the final users

18. Job Engine Collection, Parametric & Workflow
Currently the Job Engine natively support:
Collection jobs;
Parametric jobs;
N-1 workflows.
In the next releases we’ll add the support for DAG (Direct Acyclic Graph)

19. Example Job Submission
String wmsList[] = {"wms://wms-4.dir.garr.it:7443/glite_wms_wmproxy_server“,
"wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server"/};
Resource Manager List
Example Job Submission
Job Description
GEJobDescription description = new GEJobDescription();
description.setExecutable("/bin/sh");
description.setArguments("hostname.sh");
description.setInputFiles("/home/mario/Documenti/hostname.sh");
description.setOutput(“Output.txt");
description.setError(“Error.txt");
Submit the Job
JSagaJobSubmission tmpJSaga = new JSagaJobSubmission(description);
tmpJSaga.useRobotProxy("etokenserver.ct.infn.it", "8082", "332576f78a4fe70a e90cd11f", "gridit", "gridit", true);
tmpJSaga.setResourceManagerList(wmsList);
tmpJSaga.submitJobAsync("mtorrisi", " :8162", 1, "gLite Test job“);

20. Example Workflow N-1 (1/3)
InfrastructureInfo infrastructures[] = new InfrastructureInfo[2];
String wmsList[] = { "wms://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server" };
infrastructures[0] = new InfrastructureInfo("gridit", "ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList, "etokenserver.ct.infn.it", "8082“, "332576f78a4fe70a e90cd11f", "gridit", "gridit");
String globusList[] = {"wsgram://xn03.ctsf.cdacb.in:8443/GW"};
Infrastructures[1] = new InfrastructureInfo("GARUDA","wsgram","", globusList, "etokenserver.ct.infn.it","8082","332576f78a4fe70a e90cd11f","gridit","gridit");
Define Infrastructures
Example Workflow N-1 (1/3)
Job 1
Job 2
Job 3
Job N …
Final Job
OUTPUT

21. Example Workflow N-1 (2/3)
Define First Level Jobs
Example Workflow N-1 (2/3)
for (int i = 0; i < 3; i++) {
GEJobDescription description = new GEJobDescription(); description.setExecutable("/bin/sh");
switch (i) {
case 0:
description.setArguments("hostname.sh"); description.setInputFiles("/home/diego/test_wf/hostname.sh"); break;
case 1:
description.setArguments("ls.sh"); description.setInputFiles("/home/diego/test_wf/ls.sh"); break;
case 2:
description.setArguments("pwd.sh"); description.setInputFiles("/home/diego/test_wf/pwd.sh"); break; }
description.set…
descriptions.add(description);

22. Example Workflow N-1 (3/3)
GEJobDescription finalJobDescription = new GEJobDescription();
finalJobDescription.setExecutable("/bin/sh");
finalJobDescription.setArguments("ls.sh");
String tmp = "";
for(int i = 0; i < descriptions.size(); i++){
if(tmp.equals("")) tmp=descriptions.get(i).getOutput();
else tmp+=","+descriptions.get(i).getOutput(); }
finalJobDescription.setInputFiles(tmp + ",/home/diego/test_wf/ls.sh,/home/diego/test_wf/ifconfig.sh");
finalJobDescription.setOutput("myOutput-FinalJob.txt");
finalJobDescription.setError("myError-FinalJob.txt");
Define Final Job
Example Workflow N-1 (3/3)
JobCollection wf = new WorkflowN1("scardaci", "Workflow N-1", "/tmp", descriptions, finalJobDescription);
JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission(wf);
tmpJobCollectionSubmission.submitJobCollection(infrastructures, " :8162", 1);
Submit the Workflow

23. My Workspace
User Jobs
User Job Map
User’ Data
Help

24. MyJobs

25. Grid & Cloud Engine Javadoc

26. The VRC-driven GISELA Science Gateway (http://gisela-gw.ct.infn.it)
The GISELA SG is a Web site that allows users to fully exploit the e- Infrastructure both computing (jobs) and storage (data) services through a normal web browser.
Multi Languages
support

27. The VRC-driven GISELA SG: A new concept of Application Database
A new concept of Application registry allowing users to directly submit selected high impact applications on the GISELA e-infrastructure in an easy way
Links to applications run pages

28. The GISELA Science Gateway: Phylogenetics (MrBayes)
Input Parameters

29. The GISELA Science Gateway Feedback from the final review
The main technological achievement of the project was the adoption of a Science Gateway approach for the LA user communities decreasing significantly the entry barrier for new users.
The SG approach has proven to successfully answer the needs and expectations of user communities and subsequently be an attractor for Latin American stakeholders
Impact on the Latin American scientific and research community has been very good.
The technical objectives have been met and even exceeded expectations. In particular the Science Gateway approach has taken advantage of current developments and considered sensibly the demands of user communities and future e-infrastructure providers.

30. GridEEG @ DECIDE (http://applications.eu-decide.eu/)

31. The agINFRA Science Gateway R (http://aginfra-sg.ct.infn.it/)
Well known statistical data analysis tool

32. Summary and conclusions
The Catania Grid & Cloud Engine provides developers with a powerful API to port applications on a e-Infrastrucure in a esay way
The same code can be used to execute an application in different types of e-Infrastructures: Grid, Cloud, HPC, local
Several projects already adopted and successfully used the CSGF and the Grid & Cloud Engine

33. References
The Catania Science Gateway Framework:
Grid & Cloud Engine Javadoc: gateways.it/training-material
A Simple API for Grid Applications (SAGA):
JSAGA:

34. Thank you ! Would you like to evaluate the CSGF and the
Grid & Cloud Engine ?
Please, write to