DCI BRIDGE Introduction its Native Access Hands-On Akos Balasko

Outline Introduction (~10 min.) DCI-Bridge setup (~ 5 min) – Set up middlewares – View logs – Debug mode → View youtube video → utu.be Native access Hands-On (~10 min) – Precognitions Storing files Providing proxies Getting statuses – HandsOn (~ 1h) Middleware development (~ 25 min)

Introduction DCI-Bridge: – Is to submit/manage single jobs to remote resources – Supports many middlewares (from clusters till clouds) – Can be used as a stand-alone service – Provides standard interface – WS-PGRADE/gUSE uses this interface → a DCI-BRIDGE installed as a part of a normal portal instance can be invoked this way

Where are we now? 4 ASM API WS-PGRADE UI Customized UI Other, existing UI WS-PGRADE workflow UI

Check Youtube video!

Cloud Image Tomcat Storage Component representatives for hands- on Client FileServer WS-PGrade Credential Provider DCI-Bridge DCI System Status Receiver

Behind the scenes Client FileServer Credential Provider DCI-Bridge DCI System 1. store proxy 2. upload input files 3. submit JSDL 4. get proxy 5. get files 6. submit job Status Receiver

Behind the scenes Client FileServer Credential Provider DCI-Bridge DCI System 2. send status 1. send status Status Receiver

Behind the scenes Client FileServer Credential Provider DCI-Bridge DCI System 2. send status 1. send finish status Status Receiver 3. upload output files 5. download files 4. check status

Native access Hands-On Precognitions: DCI-Bridge can be invoked natively for job submission using a JSDL standard with 2 extensions. – JSDL Is a standard XML-format of job description. contains only URI's to identify file associations, not exact files – Extensions: contains status receiver web-service reference Contains Credential provider web-service reference

Example File output URI log.dlg overwrite true leUploadServlet?path= 70:8080_wspgrade/10239/AutoDock_ _ /AutoDock/outputs/balasko01/0/&am p;link=log.dlg

Pre-developed and deployed Status Receiver Service public class StatusWService implements JobStatusService { public java.lang.String sendStatus(java.lang.String pJobID, org.ggf.schemas.bes._2006._08.bes_factory.A ctivityStatusType pJobStatus, java.lang.String pResource) { System.out.println("STATUSSERVICE: Job Status arrived: jobID:" + pJobID); if(pJobStatus.getState().equals(ActivityStateEn umeration.FINISHED)) System.out.println("Status:FINISHED"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.FAILED)) System.out.println("Status:FAILED"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.RUNNING)) System.out.println("Status:RUNNING"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.PENDING)) System.out.println("Status:PENDING"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.CANCELLED)) System.out.println("Status:CANCELLED"); System.out.println("Resource: " + pResource); return new String(); }

Prepared development framework

● Create new project: File → New Project ● Select Java application ● Click Next

● Project Name: DCIBridgeInvokerDemo ● Click to Finish

● Create JAXB Binding (Right click on project name in Project Manager window, then New File) ● Click Next

Create JAX-B bindings Pick URL then type the URL (wsdl location) Mandatory and STATIC Name Pick URL then type the URL (wsdl location)

● Trust in all unknown authority (Click to Yes)

Import external libraries Right click on project Name in Project Window then select properties after it select libraries menuitem, Finally to click to add projects button Then navigate to dci-bridge_cli folder and select it And do it on information_cli folder as well

You should see something like this:

HANDS-ON section Login to your portal machine – ssh -Y Start netbeans development framework –./netbeans/bin/netbeans Open DCIBridgeInvokerDemo project's main class for editing

Edit the main method of the Main class, add the followings // Read JSDL file called "examplejob.jsdl" into a string object. // The file itself must be placed next to the jar file. File jsdlFile = new File("examplejob.jsdl"); Scanner scan = new Scanner(jsdlFile); String jsdlContent = scan.useDelimiter("\\Z").next(); //Parse the jsdl string according to the classes stored in classes array Class[] classes = new Class[]{JobDefinitionType.class, UserNameType.class, GroupNameType.class, FileNameType.class, ArgumentType.class, LimitsType.class, POSIXApplicationType.class, ExtensionType.class, SDLType.class}; JAXBContext jc = JAXBContext.newInstance(classes); Unmarshaller u = jc.createUnmarshaller(); JAXBElement obj = u.unmarshal(new StreamSource(new StringReader(jsdlContent)), JobDefinitionType.class); JobDefinitionType jsdl = (JobDefinitionType) obj.getValue();

In the main function // Submitting the jsdl file to DCI- Bridge // Create variable to store besID // (Note: in JSDL it will be the value of JobAnnotation tag) // Create object of a submitter interface class SubmitterFace client = null; // initialize submitter object as an object of hu.sztaki.lpds.dcibridge.client. SubbmitterJaxWSIMPL Class client = (SubmitterFace) Class.forName("hu.sztaki.lpds.dcibridge.client. SubbmitterJaxWSIMPL").newInstance(); // Set its main service ID client.setServiceID("/BESFactoryService?wsdl" ); // Set the web-service URL where DCI-Bridge is placed client.setServiceURL(" 080/dci_bridge_service");

In the main function // Create object of submission // (instance of CreateActivityType class) CreateActivityType submit = new CreateActivityType(); //Create a new ActivitiyDocumentType object and associate it into submit object submit.setActivityDocument(new ActivityDocumentType()); // Set the job definition to the parsed JSDL object submit.getActivityDocument().setJobDefinition(j sdl); // call client.createActivity with submit object for job submission CreateActivityResponseType rep = client.createActivity(submit); //catch exceptions

Compile the project, which produces DCIBridgeInvokerDemo.jar in DCIBridgeInvokerDemo/dist folder Copy /test/examplejob.jsdl next to the compiled jar

Edit the jsdl: (Change template placeholders to current values) CHANGE_JOB_ANNOTATION:=some unique ID CHANGE_CREDENTIALPROVIDER_URL:= CHANGE_FILESTORAGE_URL:= CHANGE_STATUSRECEIVER_URL/JobStatusS ervice?wsdl:= StatusServiceService?wsdl CHANGE_UNIQUE_FOLDER:=some unique ID

Execution Execute the application: java -jar DCIBridgeInvokerDemo.jar Check statuses in the logfile: tailf $HOME/apache-tomcat /logs/catalina.out Check output files at: $HOME/apache-tomcat /temp/storage/ 80_wspgrade/11004/AutoDock_ _ /AutoGrid/outputs/CHANGE_UNIQUE_ FOLDERNAME

End of Hands-On

Develop new middleware plugins Two scenarios available: 1.: Connect Native DCI-Bridge Service 2.: Connect whole WS-PGRADE/gUSE to a new middleware

Goal of this session + New Mid

Main architecture Admin Interface (JSP pages) Semi-autogenerated eventhandlers BES Middleware Brokering and Management Layer UnicoregLiteGT2GT4 Cloud Broker PBS Middleware Class local... JSDL

Execution with a JSDL Middleware Brokering Layer local dci-bridge host(64bit) 0 UnicoregLiteGT2GT4 Cloud Broker PBS Middleware Class local...

1.1 Creating Configuration Interface 1. Add new middleware name in mb_scheduling_description_language.xsd (other fields relevant for resource(VO,DCI) selection are generic) 2. Extend Configuration Schema with middleware-specific configuration possibilities in – dci-bridge_configuration_schema_2012.xsd (its values are stored in dcibridge.xml (e.g. place of trusted keystore ) 3. Do this in dci_bridge_service, in dcibridge_cli and in wfi components.

1.1 Creating Configuration Interface Sample extension of mb_scheduling_description_language.xsd ….. + New Mid

1.1 Creating Configuration Interface Sample dci-bridge_configuration_schema_2012.xsd

1.1 Creating Configuration Interface 2. Extend existing classes that process tab/ menu selection logic 3. Create JSPs (detailed in CreatingDCIBridgePlugin.pdf in Sourceforge)

1.2 Creating Middleware class Create the new plugin class in package hu.sztaki.lpds.submitter.grids; public class Grid_newmid extends Middleware

1.2 Creating Middleware class Implement the following methods: protected void submit(Job pJob) throws Exception; – invoked on job submission – performs job submission protected void getStatus(Job pJob) throws Exception; – invoked when job status need to be queried – queries job status and sets pJob’s status accordingly protected void abort(Job pJob) throws Exception; – invoked when the job need to be aborted – aborts the execution of the job protected void getOutputs(Job pJob) throws Exception; – invoked when the outputs of the job need to be downloaded – downloads the outputs to a local folder

1.2 Creating Middleware class You can find all data about the job (JSDL) in the pJob parameter. The input, exe and authentication files are stored by default in the job's directory: tomcat/temp/dci_bridge/jobAnnotation (set in the JSDL) The local output files mus be copied to: tomcat/temp/dci_bridge/jobAnnotation/outputs

1.2 Creating Middleware class Optional Modifications Update the configuration, if necessary: public void setConfiguration() throws Exception{} – sets/initializes additional data about the middleware

1.2 Creating Middleware class Invocation of setConfiguration() method OR at Tomcat's startup

1.2 Creating Middleware class Optional Modifications Middlewares thread: – invokes middleware methods according to the job/user interactions – handles a queue for each middleware – Default algorithm is : FIFO with put back again on demand Run method on the interpretation can be overridden, if different queue handling algorithm is required.

Where are we now? After doing these implementations, new middleware can be used by Native BES invokes of DCI-Bridge. Let's see how this middleware can be integrated and shown in WS-PGRADE/gUSE as well!

2. Modification to use the middleware in WS-PGRADE newmid

2. Create job configuration plugin (java) Create new class public class JobConfigUI_newmid implements JobConfigUI{ in package package hu.sztaki.lpds.pgportal.servlet.ajaxactions.grids; Implement the getJsp and getJobParameters methods. (Note: You can reuse the existing plugins, and you can use existing JSP pages for visualization. )

2. Create job configuration plugin (jsp) If the middleware has specific settings possibilities, a new JSP interface should be created and the control must be redirected in getJSP() method

2. Modifications in WFI component WFI generates and submits the jobs' JSDL to the DCI-BRIDGE. Modify the JobConfig class in hu.sztaki.lpds.wfi.util package: – Add the new middlewares name in mbsdlMiddleware() method. – (it maps the job's configuration to the middlewares configured in DCI-Bridge)

Thanks for your attention!