1. Development of portlets for special jobs: parametric, collections, workflows Mario Torrisi (mario.torrisi@ct.infn.it) Istituto Nazionale di Fisica Nucleare – Sezione di Cataniamario.torrisi@ct.infn.it Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA

2. 2 Outline ● Job Engine – A short recap on the Job Engine seen in the last section ● Introduction on Special Jobs – Job Collection – Parametric Job – Workflow N-1 ● Manage Special Jobs – Grid Engine API to execute Special Job on DCIs ● Portlet template – Get code from SVN – Compile and test – Customize ● Summary and conclusion

3. 3 Catania Grid & Cloud Engine Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA

4. 4 Job Engine ● The Job Engine is made of a set of libraries to develop applications able to submit and manage jobs on a grid infrastructure; ● It is compliant with the OGF SAGA standard; – JSAGA is the SAGA implementation adopted ● It is optimized to be used in a Web Portal running an application server (e.g. Glassfish, Tomcat,…) based on J2EE; ● It can be used also in stand-alone mode;

5. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 5 Job Engine Easiness ● Allows to develop applications able to submit jobs on a DCI in a very short time; ● Exposes a set of very intuitive APIs; ● Provides support for MPI applications; ● The developer has only to submit the job: – The Job Engine periodically check the job status; – When the Job is done, the job output is automatically downloaded and (if set) an email is sent to notify the the user.

6. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 6 Job Engine Features ● Middleware independent ● Easiness ● Scalability ● Performance ● Accounting ● Fault tolerance ● Workflow ● Provides easy mechanisms to support parallel applications ● JobCollection ● Parametric Job ● Workflow N-1

7. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 7 Introduction on Special Jobs Job Collection ● Job Collection: is a set of independent jobs that run in parallel ● When all jobs are successfully completed the whole collection becomes DONE Job Collection Job 1Job 2Job N...

8. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 8 Introduction on Special Jobs Parametric Job ● Parametric Job: is a set of jobs that run in parallel having the same executable, but with different arguments (e.g., input parameters) ● When all jobs are successfully completed the whole parametric job becomes DONE Parametric Job Job 1Job 2Job N...

9. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 9 Introduction on Special Jobs Workflow N-1 ● Workflow N-1 is a Special Job consisting of two different levels of jobs: – First level jobs is a set of independent jobs that run in parallel – Final (Collector) job collects all first level jobs' outputs and generates a file output ● When all first level jobs are successfully completed the final job is submitted and when it becomes DONE the whole workflow has considered completed Workflow N-1 Job 1Job 2Job N... Final Job

10. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 10 Manage Special Job ● Parametrics job and Workflows N-1 are designed as generalization of Job Collection

11. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 11 JobCollection.java ● This class handles the behaviour of the Job Collection. ● It is responsible for: – creation; – status updating; – closing of the collection when all jobs are DONE.

12. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 12 JobParametric.java ● This class handles the behavior of the Parametric Job. ● Inherits all attributes and methods defined in its parent class. ● Adds a new attribute that specify the single executable.

13. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 13 WorkflowN1.java ● This class handles the behavior of the Workflow N-1. ● Inherits all attributes and methods defined in its parent class ● Adds 3 new attributes: ● Overrides some JobCollection methods

14. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 14 Example InfrastructureInfo infrastructures[] = new InfrastructureInfo[2]; String wmsList[]={"wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server"}; String wmsList2[]={"wms://wms.magrid.ma:7443/glite_wms_wmproxy_server"}; infrastructures[0] = new InfrastructureInfo("gridit", "ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList, "etokenserver.ct.infn.it", "8082", "332576f78a4fe70a52048043e90cd11f", "gridit", "gridit"); infrastructures[1] = new InfrastructureInfo("EUMED","ldap://bdii.eumedgrid.eu:2170", wmsList2, "etokenserver.ct.infn.it", "8082", "bc681e2bd4c3ace2a4c54907ea0c379b", "eumed", "eumed"); 1. Define Infrastructures

15. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 15 Example ArrayList descriptions = new ArrayList (); for (int i = 0; i < 2; i++) { GEJobDescription description = new GEJobDescription(); description.setExecutable("/bin/sh"); switch (i) { case 0: description.setArguments("hostname.sh"); description.setInputFiles("./hostname.sh"); break; case 1: description.setArguments("ls.sh"); description.setInputFiles("./ls.sh"); break; } description.setOutputPath("/tmp"); description.setOutput("myOutput-" + i + ".txt"); description.setError("myError-" + i + ".txt"); descriptions.add(description); } 2. Define Descriptions

16. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 16 Example Job Collection submission JobCollection collection = new JobCollection("mtorrisi", "Collection - TEST - ", "/tmp", descriptions); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission("jdbc:mysql://localhost/userstracking", "tracking_user", "usertracking", collection); tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.183:8162", 10); 3. Create JobCollection object 4. Submit Job Collection

17. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 17 Example Parametric Job Submission JobCollection p = new JobParametric("test", "Parametric Job - mtorrisi", "mario.torrisi@ct.infn.it", "/tmp", descriptions, "hostname.sh"); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission( "jdbc:mysql://localhost/userstracking", "tracking_user", "usertracking", p); tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.183:8162", 10); 3. Create JobParametric object 4. Submit Parametric Job *The first two steps are common to the "Job Collection" submission

18. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 18 Example Workflow N-1 Submission (1/2) 3. Create final job description GEJobDescription finalJobDescription = new GEJobDescription(); finalJobDescription.setExecutable("ls.sh"); finalJobDescription.setArguments("./ls.sh"); String tmp = ""; for (int i = 0; i < descriptions.size(); i++) { if (tmp.equals("")) tmp = descriptions.get(i).getOutput(); else tmp += "," + descriptions.get(i).getOutput(); } finalJobDescription.setInputFiles(tmp + ",./ls.sh"); finalJobDescription.setOutputPath("/tmp"); finalJobDescription.setOutput("myOutput-FinalJob.txt"); finalJobDescription.setError("myError-FinalJob.txt"); *The first two steps are common to the "Job Collection" submission

19. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 19 Example Workflow N-1 Submission (2/2) JobCollection wf = new WorkflowN1("mtorrisi", "Workflow N-1", "/tmp", descriptions, finalJobDescription); JobCollectionSubmission tmpJobCollectionSubmission = new JobCollectionSubmission( "jdbc:mysql://localhost/userstracking", "tracking_user", "usertracking", wf); tmpJobCollectionSubmission.submitJobCollection(infrastructures, "193.206.208.183:8162", 10); 4. Create WorkFlowN1 object 5. Submit Workflow N-1

20. 20 Special Job Portlet

21. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 21 Outline ● Introductory consideration on mi- parallel-app-portlet ● Preliminary operations ● Get from SVN and deploy ● Test – Special job execution – Log extraction – MyJobs ● Customize

22. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 22 Preliminary operations ● Before testing mi-parallel-app-portlet – Install MyJobs portlet – Check the eTokenServer service is reachableeTokenServer – Check the system date is aligned with current time – Check that Grid CA certificates are updated Update CA certificates – Follow instructions to fix GridEngine 1.5.1 ● Refer to presentation seen in the last webinar to perform the above mentioned preliminary operations

23. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 23 Preliminary operations GridOperations table ● Insert a new line into the GridOperations table:

24. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 24 mi-parallel-app-portlet Get from SVN and deploy ● Get code form SVN ● Run deploy command ● Watch server.log until see something like: ● Install portlet in the portal through Liferay menu: svn checkout svn://svn.code.sf.net/p/ctsciencegtwys/liferay/trunk/gilda/mi-parallel-app-portlet mi- parallel-app-portlet ant deploy [AutoDeploy] Successfully autodeployed : /home/mario/liferay-portal-6.1.1-ce-ga2/glassfish- 3.1.2/domains/domain1/autodeploy/mi-parallel-app-portlet.

25. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 25 mi-parallel-app-portlet Test 1. Select collection type 2. Insert task number 3. Click “OK” button 4. Fill input fields 5. Insert a brief collection description 6. Submit collection

26. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 26 Special job execution Log extraction(1/2) ● Check GridEnigne.log file under /logs folder – Jobs Submission 2013-07-26 22:35:57,270 [submitJob] INFO JSagaJobSubmission - Job Submitted: [wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server]- [https://wmsdecide.dir.garr.it:9000/n_tn5247FcS3Cas4v3gfxA] […] 2013-07-26 22:36:01,046 [submitJob] INFO JSagaJobSubmission - Job Submitted: [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms024.cnaf.infn.it:9000/fnu6HvRxj71vmfyufqaslQ] […] 2013-07-26 22:36:04,531 [submitJob] INFO JSagaJobSubmission - Job Submitted: [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms024.cnaf.infn.it:9000/7lXUSv5n1em0CcrZlvQS1A] – Check job status thread: updates jobs status 2013-07-26 22:37:04,205 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job [wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server]- [https://wmsdecide.dir.garr.it:9000/n_tn5247FcS3Cas4v3gfxA] is RUNNING [...] 2013-07-26 22:37:04,934 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [https://wms024.cnaf.infn.it:9000/7lXUSv5n1em0CcrZlvQS1A] is RUNNING [...] 2013-07-26 22:37:05,060 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server][https://wms024.cnaf.infn.it:9000/fnu6HvRxj71vmfyufqa slQ] is RUNNING

27. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 27 Special job execution Log extraction(2/2) – Check job status thread: updates collection status 2013-07-26 22:37:05,697 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collections: 1 2013-07-26 22:37:05,697 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collection Status: RUNNING 2013-07-26 22:37:05,719 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - NEW Running collection Status: RUNNING – Complete collection 2013-07-26 22:40:04,258 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collection Status: RUNNING 2013-07-26 22:40:04,300 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - NEW Running collection Status: DONE 2013-07-26 22:40:04,300 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Closing Done Collection: JobCollection [id=67, description=Demo Collection: 27/7/2013 - 0:35:33, commonName=test, taskCounter=3] 2013-07-26 22:40:04,301 [pool-31-thread-1] INFO JobCollection - creating a tgz archive containing output files... [...] 2013-07-26 22:40:04,301 [pool-31-thread-1] INFO JobCollection - tar czvf /tmp/jobOutput/DemoCollection277201303533_67.tgz --directory=/tmp/jobOutput/ DemoCollection277201303533_67

28. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 28 MyJobs Collection status Jobs current status Collection DONE

29. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 29 Customize ● Customize.sh is a BASH script to clone mi-parallel-app-portlet ● Make a copy of mi-parallel-app-portlet in _portlet – cp -R mi-parallel-app-portlet _portlet ● Move into _portlet folder and edit customize.sh – AUTH_EMAIL= your@email – AUTH_NAME=' ' – AUTH_INSTITUTE=' ' – APP_OPERATIONID= – APP-NAME= ● Run customize script –./cusotmize.sh

30. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 30 Summary and conclusion ● Catania Grid & Cloud Engine provides powerful APIs to port applications on e-Infrastructure easily ● Developers do not need to care of the differences between the middleware details ● Developers have to only prepare the job(s) description(s) and then to submit it(them) ● The JobEngine: – Periodically checks the status – Automatically downloads the outputs and notify the user ● Job Engine provides also powerful mechanisms to guarantee jobs submission (or re-submission), if they fail due infrastructure issues

31. Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 31 References ● Catania Science Gateway Framework http://www.catania-science-gateways.it/ http://www.catania-science-gateways.it/ ● Grid & Cloud Engine Javadoc http://grid.ct.infn.it/webinar-liferay/ http://grid.ct.infn.it/webinar-liferay/ ● Simple API for Grid Applications (SAGA) http://www.ogf.org/documents/GFD.90.pdf http://www.ogf.org/documents/GFD.90.pdf ● JSaga: Java implementation of SAGA specification http://grid.in2p3.fr/jsaga/ ● Training material https://gilda.ct.infn.it/wikimain

32. 32 Thank you!