Job Management Portlet Mario Torrisi Istituto Nazionale di Fisica Nucleare – Sezione di Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA

Outline Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 2 Catania - 12 Marzo 2014  Pre-requisites  MyJobs portlet Installation  GridOperations table  Portlet Template  Get from SVN  Compile and test  Job submit logging extraction  Code explanations  init() method and preferences  The ACTION/VIEW enums and the portletStatus variable  The AppInput Class  The getInputForm method  The submitJob method  Preferences

MyJobs(1/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 3  MyJob war file available from SourceForge at:  MYJOBSURL= gateway/wars/1307/MyJobs.war/download  curl -L $MYJOBSURL -o MyJobs.war  or, wget $MYJOBSURL -O MyJobs.war  Deploy MyJobs with: cp MyJobs $LIFERAY_HOME/deploy/  Watch the Liferay’ server.log file till:  ‘MyJobs successfully deployed’  Install the portlet into the portal through liferay menu: Catania - 12 Marzo 2014

MyJobs (2/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 4 Catania - 12 Marzo 2014 Active Jobs Job Status and Output Search on job description Job description

GridOperations table (1/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 5 Catania - 12 Marzo 2014  Each user action which involves the distributed infrastructure will be tracked by the UsersTracking Database  The GridEngine uses the GridOperations table to register applications and services accessing the distributed infrastructure mysql -u tracking_user -pusertracking userstracking desc GridOperations; | Field | Type | Null | Key | Default | Extra | | id | int(11) | NO | PRI | NULL | auto_increment | | portal | varchar(120) | NO | | NULL | | | description | varchar(200) | NO | | NULL | |  We can register and then track the activity of our developed portlet with: insert into GridOperations values (9,'test','testPortlet');

GridOperations table (2/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 6 Catania - 12 Marzo 2014  How to get GridOperations values  id – Just a numeric value; ‘9’ historically used by Tester Apps  portal – Use the value highlighted in the figure; Liferay’ right, top-most menu:  description – Use any human readable application description  Application registration in the GridOperations table is mandatory for the MyJobs portlet  GridOperations values will be carefully selected for production portals

svn checkout hostname-portlet Portlet template Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 7 Catania - 12 Marzo 2014  The portlet template is a complete example of JSR 286 compliant portlet able to submit a sequential Job into a distributed environment.  Its java code extends the GenericPortlet class and uses JSP pages to generate the input GUI  Java code contains java-doc compliant remarks able to generate automatically help pages

Portlet template Features Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 8 Catania - 12 Marzo 2014  It provides a full example on:  Define application default values  Define distributed infrastructure settings where the application will run  Manage input elements from web forms as application input  Manage upload file requests  Manage portlet preferences to handle distributed infrastructure settings  View/Manage the ‘pilot script’ which contains the batch instructions that will be executed on the remote Worker Node (WN).  The portlet template can be ‘customized’ as shown by the previous presentation on the ‘generic’ portlet.  Make a full copy of the mi-hostname-portlet  cp mi-hostname-portlet mi-custom-portlet  Open the customize.sh script inside the mi-custom-portlet directory and change all configurable parameters, then execute it: ./customize.sh  The new portlet is now an hostname clone configured with your settings and ready to be further customized for application specific requirements

Portlet template Install & deploy Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 9 Catania - 12 Marzo 2014  Compile the template portlet  cd mi-hostname-portlet  ant deploy (then watch the Liferay’ server log file)  Deploy the portlet from Liferay menu as already done for MyJobs  Location: Add/More/GILDA/mi-hostname-portlet  Test the portlet  Press the ‘Demo’ and then ‘Submit’ buttons  It is recommended to watch the Liferay’ server log file during the whole process Bottom Top

Portlet template Meaning & usage Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 10 Catania - 12 Marzo 2014  mi-hostname-portlet  mi – Multi Infrastructure; it allows to execute the same application on many different insfrastructures (gLite, only)  A multi-Infrastructure+multi-Middleware example exists as well and currently running on Chain project Science Gateway. Its code available on Sourceforge as well.  hostname – The execution of the ‘hostname’ command on the distributed environments is used exactly like the ‘Helloworld’ examples while approaching new programming languages.  It accepts an input file or a text inside the bigger text-area in mutually exclusive fashion. Then just specify a job description string and submit the job with the ‘Submit’ button

Before testing the job execution Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 11 Catania - 12 Marzo 2014  Open the VPN or be sure the eTokenserver allows incoming connections on port 8082 form your portal IP address  Check the eTokenserver service is reachable  Connect to  From the interface generate the robot proxy request  Execute curl or wget on the generated request  Check the system date is aligned with current time  On linux check the ntpd service; re-align date as root with: /etc/init.d/ntpd stop ; ntpdate ntp-1.infn.it ; ntpd start  Check that Grid CA certificates are updated  curl > grid_settings.tar.gz  tar xvfz grid_settings.tar.gz -C /etc/grid-security/

Job execution Log extraction (1/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 12 Catania - 12 Marzo 2014  Liferay server.log file reports:  A full dump of the portlet preference values  The GridEngine Initialization  The Robot proxy retrieval [mi_hostname_portlet:108] dump: Infrastructure #1 enableInfrastructure : 'yes' nameInfrastructure : 'EUMEDGRID-Support infrastructure' acronymInfrastructure: 'EUMEDGRID' [mi_hostname_portlet:108] dump: Infrastructure #1 enableInfrastructure : 'yes' nameInfrastructure : 'EUMEDGRID-Support infrastructure' acronymInfrastructure: 'EUMEDGRID' INFO JSagaJobSubmission - Getting adaptor name... JSagaJobSubmission - Using adaptor: wms INFO RobotProxy - proxyPath=/tmp/7f7e1e98-0fd1-4ebb-a1ae-0627efddf600 INFO RobotProxy - get proxy: 043e90cd11f?voms=gridit:gridit&proxy-renewal=true INFO RobotProxy - proxyPath=/tmp/7f7e1e98-0fd1-4ebb-a1ae-0627efddf600 INFO RobotProxy - get proxy: 043e90cd11f?voms=gridit:gridit&proxy-renewal=true

Job execution Log extraction (2/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 13 Catania - 12 Marzo 2014  The JSAGA job submission string  The input sandbox file transfers  The job id and the job status thread execution INFO JSagaJobSubmission - jobSandbox:/opt/liferay-portal ce- ga2/glassfish-3.1.2/domains/domain1/autodeploy/mi-hostname-portlet/WEB- INF/job/pilot_script.sh>pilot_script.sh,/tmp/ _test_input_file. txt> _test_input_file.txt,/tmp//jobOutput/multiinfrastructurede mojobdescription1_1/<hostname- Output.txt,/tmp//jobOutput/multiinfrastructuredemojobdescription1_1/<hostnam e-Error.txt Connecting to Gsiftp service at: wms014.cnaf.infn.it: JSagaJobSubmission - Job Submitted: [wms://wms014.cnaf.infn.it:7443/glite_wms_wmproxy_server]- [ UsersTrackingDBInterface - UpdateJobsStatusAsync running in Thread : Thread[pool-103-thread-1,5,grizzly-kernel]

mi-hostname-portlet Code structure Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 14 Catania - 12 Marzo 2014 docroot/ css/ main.css images/ AppLogo.png lib/ submit.jsp viewPilot.jsp edit.jsp help.jsp input.jsp icon.png js/ main.js WEBINF/ (See next slide) build.xml customize.sh css styles definition file Application logo Portlet libraries from third parties JSP pages for: View,Help,Edit portlet modes JSP pages for: View,Help,Edit portlet modes WEBINF Contains portlet configurations, Grid job definition and portlet Java code WEBINF Contains portlet configurations, Grid job definition and portlet Java code Ant build definition file Customization script

mi-hostname-portlet Code structure Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 15 Catania - 12 Marzo 2014 WEB-INF/ classes/ src/it/infn/ct/ AppInfrastructureInfo.java AppLogger.java AppPreferences.java mi_hostname_portlet.java tld/ job/ standard_pilot_script.sh pilot_script.sh liferay-display.xml liferay-portlet.xml glassfish-web.xml portlet.xml css styles definition file Portlet’ Java code Portlet’ XML configuration files XML semantic Grid Job files

mi-hostname-portlet Source code Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 16 Catania - 12 Marzo 2014  Source code available form SourceforgeSourceforge  3 Classes:  AppInfrastructureInfo.java  Maintains the information about the distributed infrastructure resources and services  AppLogger.java  Wrapper class dedicated to the logging activity  AppPreferences.java  Contains portlet preferences related to the Grid job and the distributed infrastructure resource settings  mi_hostname_portlet.java  Main portlet code which inherits form the GenericPortlet class. It manages the portlet GUI and instructs the GridEngine to submit the job.

mi-hostname-portlet Type & Classes Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 17 Catania - 12 Marzo 2014  Actions enumerated types ACTION_INPUT // User asked to submit a job,ACTION_SUBMIT // User asked to rerutn to the input form,ACTION_PILOT // The user did something in the edit pilot screen pane  Views enumerated types VIEW_INPUT // View containing application input fields,VIEW_SUBMIT // View reporting the job submission,VIEW_PILOT // Shows the pilot script and makes it editable  The couple: ( PortletMode, PortletStatus ) determines the entire portlet workflow; both values handled by JSP pages and Java code. The second variable used both for processAction and doView

AppInput class Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 18 Catania - 12 Marzo 2014  AppInput class defined inside the main Java code source  This class is ment to collect all application GUI inputs and must be dynamically instanciated inside the processAction as soon as the user press the ‘Submit’ button class AppInput { // Applicatoin inputs String inputFileName; // Filename for application input file String inputFileText; // GUI Textfield content String jobIdentifier; // User' given job identifier // Each inputSandobox file must be declared below // This variable contains the content of an uploaded file String inputSandbox_inputFile; // Some user level information // must be stored as well String username; String timestamp;

processAction Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 19  Called when the user perform an action on the UI  Get portal and user information: portalName, username  From the request object retrieve the PortletMode; an if … else if … chain handles each different mode: EDIT, VIEW, HELP  VIEW  Gets the PortletStatus value (Actions) and use this value inside the switch() statement: ACTION_INPUT, ACTION_SUBMIT, ACTION_PILOT  EDIT  Manages the portlet preferences interface controls  HELP  Does nothing; just Logs the state  ProcessAction sends parameters to the doView() method through the response object Catania - 12 Marzo 2014 processAction(ActionRequest request, ActionResponse response)

doView Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 20 Catania - 12 Marzo 2014 doView(RenderRequest request, ResderResponse response)  From request object retrieve the PortletStatus value and determine the variable value currentView and using it for a switch statement  Inside the switch, just select the proper jsp page case VIEW_SUBMIT: { _log.info("VIEW_SUBMIT Selected..."); String jobIdentifier = request.getParameter("jobIdentifier"); request.setAttribute("jobIdentifier", jobIdentifier); PortletRequestDispatcher dispatcher=getPortletContext().\ getRequestDispatcher("/submit.jsp"); dispatcher.include(request, response); } case VIEW_SUBMIT: { _log.info("VIEW_SUBMIT Selected..."); String jobIdentifier = request.getParameter("jobIdentifier"); request.setAttribute("jobIdentifier", jobIdentifier); PortletRequestDispatcher dispatcher=getPortletContext().\ getRequestDispatcher("/submit.jsp"); dispatcher.include(request, response); } // Different actions will be performed accordingly to the // different possible view modes switch(Views.valueOf(currentView)) { … // Different actions will be performed accordingly to the // different possible view modes switch(Views.valueOf(currentView)) { …

VIEW_INPUT VIEW mode workflow Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 21 Catania - 12 Marzo 2014 ACTION_INPUT  VIEW_INPUT ACTION_SUBMIT  VIEW_SUBMIT

EDIT mode workflow Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 22 Catania - 12 Marzo 2014 doEdit() pref_action : next, previous, add, remove, viewPilot pref_action : next, previous, add, remove, viewPilot VIEW_PILOT VIEW_INPUT

Variable: JAVA  JSP Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 23 Catania - 12 Marzo 2014  From doView/doEdit/doHelp:  Inside the JSP refer the variable with:  id: Variable name  class: java.lang.String (or any other class)  scope: Normally set to ‘ request ’  Inside JSP code refer the variable value with: request.setAttribute(” ", " "); " class=" " scope=”var scope"/> %>

Variable: JSP  JAVA Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 24 Catania - 12 Marzo 2014  Inside the JSP use values inside s :  Inside the Java code refer the variable with (ProcessAction) :  Above example works for ‘normal’ s  mi-hostname-porltet uses a particular kind of  ‘multipart’ allows file upload but requires to manually handle input parameters as done by the getInputForm() method " id=" " … > (String)request.getParameter(" ");

getInputForm() (1/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 25 Catania - 12 Marzo 2014  Necessary to handle file uploads using the apache’ commons.io.* libraries  Mainly consists of a loop which set the java variables  It uses an enumerated type to identify input controls  Enter the input parameters extraction loop: private enum inputControlsIds { // Input control’ name,… // Other controls }; if (PortletFileUpload.isMultipartContent(request))

getInputForm() (2/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 26 Catania - 12 Marzo 2014  The loop: FileItemFactory factory = new DiskFileItemFactory(); PortletFileUpload upload = new PortletFileUpload( factory ); List items = upload.parseRequest(request); File repositoryPath = new File("/tmp"); DiskFileItemFactory diskFileItemFactory = new DiskFileItemFactory(); diskFileItemFactory.setRepository(repositoryPath); Iterator iter = items.iterator(); while (iter.hasNext()) { FileItem item = (FileItem)iter.next(); String fieldName =item.getFieldName(); String fileName =item.getName(); String contentType=item.getContentType(); boolean isInMemory =item.isInMemory(); long sizeInBytes=item.getSize(); switch(inputControlsIds.valueOf(fieldName)) { case var_name: appInput. =item.getString(); break;

processInputFile Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 27 Catania - 12 Marzo 2014  Called in case of input files:  processInputFile sets the appInput’ text variable with the uploaded file content (just an example)  mi-hostname-portlet has two utilities:  WARNING: File content is logged; disable this for binary files processInputFile(FileItem item, AppInput appInput) { File uploadedFile = new File(theNewFileName); try { item.write(uploadedFile); } private String updateString(String file) {…} private void storeString(String fileName,String fileContent) {…}

submitJob (1/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 28  Inside the processAction: Catania - 12 Marzo 2014 // Get current preference values getPreferences(request,null); // Create the appInput object AppInput appInput = new AppInput(); // Stores the user submitting the job appInput.username=username; // Determine the submissionTimeStamp SimpleDateFormat dateFormat = new SimpleDateFormat(tsFormat); String timestamp = dateFormat.format(Calendar.getInstance().getTime()); appInput.timestamp=timestamp; // Process input fields and files to upload getInputForm(request,appInput); // Following files have to be updated with // values taken from textareas or from uploaded files: // input_file.txt updateFiles(appInput); // Submit the job submitJob(appInput);

submitJob (2/2) Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 29  Create GEJobDescription object:  Actual job submission Catania - 12 Marzo 2014 GEJobDescription description = new GEJobDescription(); description.setExecutable("/bin/sh"); description.setArguments("hostname.sh"); description.setInputFiles(…); description.setOutputPath(….); description.setOutput(…); description.setError(…); miJobSubmission = new MultiInfrastructureJobSubmission(description); // or for development environments // MultiInfrastructureJobSubmission(,description); miJobSubmission.submitJobAsync(appInput.username, portalIPAddress, applicationId, appInput.jobIdentifier);

Prefereces Generic Settings Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 30 Catania - 12 Marzo 2014  Log level  Define different levels for the logging activity: INFO, WARN, ERROR,…  GridOperations id table value  Number of insfrastructures  GridOperations description  Application requirements ‘Job Requirements’  Pilot job, shows the batch that will be executed remotely  UsersTrackingDB connection settings. When !NULL, the portlet will operate in DEBUG mode

Prefereces Infrastructure settings Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 31 Catania - 12 Marzo 2014  Enable Infrastructure  TRUE/FALSE flag  Infrastructure number shown  Name of the infrastructure  Infrastructure short name  Information System service  WMS (optional)  GridEngine allow to specify even a list of Computing Elements; this is not handled by the example  RobotProxy settings (Host/Port/VO/Roles/Renewal flag)  LocalProxy (Used to bypass the eTokenserver; using Personal Certificates)  Software Tags specify which resource is capable to satisfy the tag request  WARNING: MultiMiddleware example overrides the meaning of some fields; especially the BDII host which identifies the JSAGA adaptor

Prefereces The code Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 32 Catania - 12 Marzo 2014  AppPreferences.java  Class that contains all portlet’ preference values  It provides a method that returns the GridEngine’ InfrastructureInfo object  The portlet init() method loads default settings from the portlet.xml file (customize.sh creates dynamically this file)  The init() method creates two object:  AppInitPreferences – Containing default values taken from portlet.xml  AppPreferences – Containing portal administrator’ settings (if changed)  AppInitPreferences – Meant to revert default settings (not implemented)  The customize.sh portlet allow to define default settings for many infrastructures

Customize.sh Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 33 Catania - 12 Marzo 2014  BASH script to clone the mi-hostname-portlet, but renaming the whole project with your specific application details, just changing few environment variables  Essential customization values  AUTH_* values, meant for Application Author details  APP_OPERDESC GridOperation’ description field (optional)  APP_OPERATIONID The GridOperation.Id for this application ( 9 default)  APP_NAME The Application name  APP_VERSION The version number of your application  INIT_NUMINFRASTRUCTURES The number of infrastructures  INIT_n_ Set of infrastructure values  1 <= n <= INIT_NUMINFRASTRUCTURES

References Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA 34 Catania - 12 Marzo 2014  GridEngine JavadocJavadoc  M.I.JobSubmission ConstructorsConstructors  SVN mi-hostname-portletmi-hostname-portlet  SVN helloworld-portlet (MM/MI) examplehelloworld-portlet

Thank you ! Catania - 24 June 2013 Webinar on the DCH-RP e-Culture Science Gateway 35 For more information on Catania Science Gateway framework please visit: