1. DCI BRIDGE Introduction its Native Access Hands-On Akos Balasko balasko@sztaki.mta.hu

2. Outline Introduction (~10 min.) DCI-Bridge setup (~ 5 min) – Set up middlewares – View logs – Debug mode → View youtube video → http://www.youtube.com/watch?v=6FyAvGt6dG4&feature=yo utu.be Native access Hands-On (~10 min) – Precognitions Storing files Providing proxies Getting statuses – HandsOn (~ 1h) Middleware development (~ 25 min)

3. Introduction DCI-Bridge: – Is to submit/manage single jobs to remote resources – Supports many middlewares (from clusters till clouds) – Can be used as a stand-alone service – Provides standard interface – WS-PGRADE/gUSE uses this interface → a DCI-BRIDGE installed as a part of a normal portal instance can be invoked this way

4. Where are we now? 4 ASM API WS-PGRADE UI Customized UI Other, existing UI WS-PGRADE workflow UI

5. Check Youtube video!

6. Cloud Image Tomcat Storage Component representatives for hands- on Client FileServer WS-PGrade Credential Provider DCI-Bridge DCI System Status Receiver

7. Behind the scenes Client FileServer Credential Provider DCI-Bridge DCI System 1. store proxy 2. upload input files 3. submit JSDL 4. get proxy 5. get files 6. submit job Status Receiver

8. Behind the scenes Client FileServer Credential Provider DCI-Bridge DCI System 2. send status 1. send status Status Receiver

9. Behind the scenes Client FileServer Credential Provider DCI-Bridge DCI System 2. send status 1. send finish status Status Receiver 3. upload output files 5. download files 4. check status

10. Native access Hands-On Precognitions: DCI-Bridge can be invoked natively for job submission using a JSDL standard with 2 extensions. – JSDL Is a standard XML-format of job description. contains only URI's to identify file associations, not exact files – Extensions: contains status receiver web-service reference Contains Credential provider web-service reference

11. Example File output URI log.dlg overwrite true http://192.168.143.170:8080/storage/Fi leUploadServlet?path=http:__192.168.143.1 70:8080_wspgrade/10239/AutoDock_2012- 11-12-085005_2012-11-13- 165752/AutoDock/outputs/balasko01/0/&am p;link=log.dlg

12. Pre-developed and deployed Status Receiver Service public class StatusWService implements JobStatusService { public java.lang.String sendStatus(java.lang.String pJobID, org.ggf.schemas.bes._2006._08.bes_factory.A ctivityStatusType pJobStatus, java.lang.String pResource) { System.out.println("STATUSSERVICE: Job Status arrived: jobID:" + pJobID); if(pJobStatus.getState().equals(ActivityStateEn umeration.FINISHED)) System.out.println("Status:FINISHED"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.FAILED)) System.out.println("Status:FAILED"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.RUNNING)) System.out.println("Status:RUNNING"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.PENDING)) System.out.println("Status:PENDING"); else if(pJobStatus.getState().equals(ActivityStateEn umeration.CANCELLED)) System.out.println("Status:CANCELLED"); System.out.println("Resource: " + pResource); return new String(); }

13. Prepared development framework

14. ● Create new project: File → New Project ● Select Java application ● Click Next

15. ● Project Name: DCIBridgeInvokerDemo ● Click to Finish

16. ● Create JAXB Binding (Right click on project name in Project Manager window, then New File) ● Click Next

17. Create JAX-B bindings Pick URL then type the URL (wsdl location) Mandatory and STATIC Name Pick URL then type the URL (wsdl location)

18. ● Trust in all unknown authority (Click to Yes)

19. Import external libraries Right click on project Name in Project Window then select properties after it select libraries menuitem, Finally to click to add projects button Then navigate to dci-bridge_cli folder and select it And do it on information_cli folder as well

20. You should see something like this:

21. HANDS-ON section Login to your portal machine – ssh -Y lpds@Your_Image_IP Start netbeans 6.5.1 development framework –./netbeans/bin/netbeans Open DCIBridgeInvokerDemo project's main class for editing

22. Edit the main method of the Main class, add the followings // Read JSDL file called "examplejob.jsdl" into a string object. // The file itself must be placed next to the jar file. File jsdlFile = new File("examplejob.jsdl"); Scanner scan = new Scanner(jsdlFile); String jsdlContent = scan.useDelimiter("\\Z").next(); //Parse the jsdl string according to the classes stored in classes array Class[] classes = new Class[]{JobDefinitionType.class, UserNameType.class, GroupNameType.class, FileNameType.class, ArgumentType.class, LimitsType.class, POSIXApplicationType.class, ExtensionType.class, SDLType.class}; JAXBContext jc = JAXBContext.newInstance(classes); Unmarshaller u = jc.createUnmarshaller(); JAXBElement obj = u.unmarshal(new StreamSource(new StringReader(jsdlContent)), JobDefinitionType.class); JobDefinitionType jsdl = (JobDefinitionType) obj.getValue();

23. In the main function // Submitting the jsdl file to DCI- Bridge // Create variable to store besID // (Note: in JSDL it will be the value of JobAnnotation tag) // Create object of a submitter interface class SubmitterFace client = null; // initialize submitter object as an object of hu.sztaki.lpds.dcibridge.client. SubbmitterJaxWSIMPL Class client = (SubmitterFace) Class.forName("hu.sztaki.lpds.dcibridge.client. SubbmitterJaxWSIMPL").newInstance(); // Set its main service ID client.setServiceID("/BESFactoryService?wsdl" ); // Set the web-service URL where DCI-Bridge is placed client.setServiceURL("http://192.168.143.170:8 080/dci_bridge_service");

24. In the main function // Create object of submission // (instance of CreateActivityType class) CreateActivityType submit = new CreateActivityType(); //Create a new ActivitiyDocumentType object and associate it into submit object submit.setActivityDocument(new ActivityDocumentType()); // Set the job definition to the parsed JSDL object submit.getActivityDocument().setJobDefinition(j sdl); // call client.createActivity with submit object for job submission CreateActivityResponseType rep = client.createActivity(submit); //catch exceptions

25. Compile the project, which produces DCIBridgeInvokerDemo.jar in DCIBridgeInvokerDemo/dist folder Copy /test/examplejob.jsdl next to the compiled jar

26. Edit the jsdl: (Change template placeholders to current values) CHANGE_JOB_ANNOTATION:=some unique ID CHANGE_CREDENTIALPROVIDER_URL:= http://YOUR_IMAGE_IP:8080/wspgrade CHANGE_FILESTORAGE_URL:= http://YOUR_IMAGE_IP:8080/storage CHANGE_STATUSRECEIVER_URL/JobStatusS ervice?wsdl:= http://YOUR_IMAGE_IP:8080/StatusService/Job StatusServiceService?wsdl CHANGE_UNIQUE_FOLDER:=some unique ID

27. Execution Execute the application: java -jar DCIBridgeInvokerDemo.jar Check statuses in the logfile: tailf $HOME/apache-tomcat- 6.0.35/logs/catalina.out Check output files at: $HOME/apache-tomcat- 6.0.35/temp/storage/http:__192.168.152.54:80 80_wspgrade/11004/AutoDock_2012-11-12- 085005_2012-11-26- 165432/AutoGrid/outputs/CHANGE_UNIQUE_ FOLDERNAME

28. End of Hands-On

29. Develop new middleware plugins Two scenarios available: 1.: Connect Native DCI-Bridge Service 2.: Connect whole WS-PGRADE/gUSE to a new middleware

30. Goal of this session + New Mid

31. Main architecture Admin Interface (JSP pages) Semi-autogenerated eventhandlers BES Middleware Brokering and Management Layer UnicoregLiteGT2GT4 Cloud Broker PBS Middleware Class local... JSDL

32. Execution with a JSDL Middleware Brokering Layer local dci-bridge host(64bit) 0 UnicoregLiteGT2GT4 Cloud Broker PBS Middleware Class local...

33. 1.1 Creating Configuration Interface 1. Add new middleware name in mb_scheduling_description_language.xsd (other fields relevant for resource(VO,DCI) selection are generic) 2. Extend Configuration Schema with middleware-specific configuration possibilities in – dci-bridge_configuration_schema_2012.xsd (its values are stored in dcibridge.xml (e.g. place of trusted keystore ) 3. Do this in dci_bridge_service, in dcibridge_cli and in wfi components.

34. 1.1 Creating Configuration Interface Sample extension of mb_scheduling_description_language.xsd ….. + New Mid

35. 1.1 Creating Configuration Interface Sample dci-bridge_configuration_schema_2012.xsd

36. 1.1 Creating Configuration Interface 2. Extend existing classes that process tab/ menu selection logic 3. Create JSPs (detailed in CreatingDCIBridgePlugin.pdf in Sourceforge)

37. 1.2 Creating Middleware class Create the new plugin class in package hu.sztaki.lpds.submitter.grids; public class Grid_newmid extends Middleware

38. 1.2 Creating Middleware class Implement the following methods: protected void submit(Job pJob) throws Exception; – invoked on job submission – performs job submission protected void getStatus(Job pJob) throws Exception; – invoked when job status need to be queried – queries job status and sets pJob’s status accordingly protected void abort(Job pJob) throws Exception; – invoked when the job need to be aborted – aborts the execution of the job protected void getOutputs(Job pJob) throws Exception; – invoked when the outputs of the job need to be downloaded – downloads the outputs to a local folder

39. 1.2 Creating Middleware class You can find all data about the job (JSDL) in the pJob parameter. The input, exe and authentication files are stored by default in the job's directory: tomcat/temp/dci_bridge/jobAnnotation (set in the JSDL) The local output files mus be copied to: tomcat/temp/dci_bridge/jobAnnotation/outputs

40. 1.2 Creating Middleware class Optional Modifications Update the configuration, if necessary: public void setConfiguration() throws Exception{} – sets/initializes additional data about the middleware

41. 1.2 Creating Middleware class Invocation of setConfiguration() method OR at Tomcat's startup

42. 1.2 Creating Middleware class Optional Modifications Middlewares thread: – invokes middleware methods according to the job/user interactions – handles a queue for each middleware – Default algorithm is : FIFO with put back again on demand Run method on the interpretation can be overridden, if different queue handling algorithm is required.

43. Where are we now? After doing these implementations, new middleware can be used by Native BES invokes of DCI-Bridge. Let's see how this middleware can be integrated and shown in WS-PGRADE/gUSE as well!

44. 2. Modification to use the middleware in WS-PGRADE newmid

45. 2. Create job configuration plugin (java) Create new class public class JobConfigUI_newmid implements JobConfigUI{ in package package hu.sztaki.lpds.pgportal.servlet.ajaxactions.grids; Implement the getJsp and getJobParameters methods. (Note: You can reuse the existing plugins, and you can use existing JSP pages for visualization. )

46. 2. Create job configuration plugin (jsp) If the middleware has specific settings possibilities, a new JSP interface should be created and the control must be redirected in getJSP() method

47. 2. Modifications in WFI component WFI generates and submits the jobs' JSDL to the DCI-BRIDGE. Modify the JobConfig class in hu.sztaki.lpds.wfi.util package: – Add the new middlewares name in mbsdlMiddleware() method. – (it maps the job's configuration to the middlewares configured in DCI-Bridge)

48. Thanks for your attention!