Remote Api Tutorial How to call WS-PGRADE workflows from remote clients through the http protocol?

Goals - Solutions Submission a workflow from a http client (program environment) avoiding the interactive (human) Portal layer of the WS-Pgrade infrastructure. Solution: The remote api web service extension of the WS-Pgarde infrastructure will be used as application layer in order to communicate with the backend (gUSE) instead of the Portal. The workflows to be submit should be modifiable by the caller. – As the WS-PGRADE workflow definitions contain own set of input files, first of all the input files are candidates to be changed but the files of job executable, and the command line parameters can be changed as well. Solution: Manual change of files to be replaced and the creation of new association table (coded in the „portmapping.txt” file). The main workflow descriptor file.xml should be modified only in the case when command line parameters belonging to the individual nodes(jobs) of the workflow must be modified. On completion of a workflow submission session the common resources should be cleared. Solution: The servers side data belonging to the workflow (submission) will be cleared upon the successful execution of the „Get output” command of the client

Typical workflow development scenario in case of remote call WS PGRADE SERVER 1 Portal gUSE DCI Bridge Browser GRAPH Ed Orig user files.zip.xml tree of files WS PGRADE SERVER 2 Portal gUSE Remote API Serv. DCI Bridge.xml.zip portmapping.txt HTTP Client Copy (modify) Upd. user files Copy and rename Eventual update Set of computational resources 1.Create and submit original workflow 2. Download tested workflow 3. Reengineering of wf definition 4. Submit the modified workflow Client side Server side

Typical workflow development scenario in case of remote call WS PGRADE SERVER 1 Portal gUSE DCI Bridge.xml.zip portmapping.txt HTTP Client Copy (modify) Upd. user files Copy and rename Eventual update Set of computational resources WS PGRADE SERVER 2 Portal gUSE Remote API Serv. DCI Bridge Browser GRAPH Ed Orig user files 1.Create and submit original workflow.zip.xml tree of files 2. Download tested workflow 3. Reengineering of wf definition 4. Submit the modified workflow Client side Server side

Summary of workflow development and remote submission process: 1.Download the original workflow from the WS-Pgrade gUSE server. 2.Make a little reengineering: – separate the structure description from the other parts – explore the user defined (not channel) input files connected to the numbered ports of the named jobs and the files of executable of the named jobs – Pack the needed files together and create a text file which describes the association of the files and the named jobs and ports 3.Create the needed script files needed to habitual submission process (call, observe, download (delete)) 4.Execute the scripts on the client machine

Basic assumptions of remote call setup There is a working WS-Pgrade workflow engine (server) where the Remote API Servlet* extension has been installed There is a client machine containing the description (structure, input files) of a workflow The client can reach the server by the http protocol There is a sever created password known by the client(s) The client has the necessary proxy certificate file if the submissions involved in the workflow are directed to such resources which need certificate bound authorization * (see installation file and documentation on Sourceforge for version : Code: /remote-3.4.tgz/download remote-3.4.tgz/download Documentation: /Documentation/RemoteAPI_Install_Manual.pdf/downlo ad ) Documentation/RemoteAPI_Install_Manual.pdf/downlo ad

The reconstruction and submission of a working „AutoDoc” PS workflow will be discussed in this tutorial. For the sake of simplicity in this case the selected submission resource does not require a certificate bound authorization and the original files will not be changed. Part II: Step by step tutorial

1a. Analysis of the „AutoDoc” workflow We have to search for the five „free” input files of the job „AutoGrid” and for the executables of the jobs „AutoGrid”, „AutoDoc” and „Collector” We have to search for the five „free” input files of the job „AutoGrid” and for the executables of the jobs „AutoGrid”, „AutoDock” and „Collector”

1b. Download the workflow „AutoDoc” from the server

Structure of the downloaded „AutoDoc” workflow 1. level: The compressed file 2. level: The needed „workflow.xml” file and the root of file structure 3. level: The list of jobs 4. level: Enumeration of input ports of a given job Standard name of job executable files „0” is the standard name of input file associated to the given port

2 Reengineering 1.Find the „workflow.xml” file 2.Rename, gather and compress the „0”named input port files, and the „execute.bin” named job executable files. 3. Make a file named „portmapping.txt” contains lines of the following syntax: = / / OR = / Notes: In section 3. use the new names defined in section2. The new file names should be different. As the separator of lines within the file „portmapping.txt” is the unix compatible linefeed character (´\n´) do not edit this file in Windows - or do it there with care.

Reengineering steps 1 and 2a (copy and rename)

Reengineering 2.2b Compress the found user defined workflow files We have the two of the three needed files: „inputs.zip” and „workflow.xml” We have two of the three needed files: „inputs.zip” and „workflow.xml”

2.3 Reengineering: Creation of the „portmapping.txt” file

3.1.a Preparation of submission Structure of the „curl” command curl { –F = } If the is preceeded by the charcter then it indicates the path of a file which must be transferred to the server and renamed as

3.1.b The structure of the Api maps the submission session: The leading workflow submission command („submit”) returns a process identifier, the other commands („info” – workflow status „detailsinfo” – detailed workflow status „download” - get workflow output after finish „stop” - delete the suspended workflow „suspend” - suspend the run of the workflow „rescue” - continues the suspended workflow) are referencing the process identifier

Submit Command example: „LocalSubmit.sh” Workflow descriptor files URL of server containing the requested service Value of password „m” indicates the subcommand selector interpreted by the „RemoteServlet”

Server side placement of password

Status observation command example: „Status.sh” Process identifier (result of the submit command) is expected here – in this case - as command line argument

Download Output command example: „GetOutput.sh” The requested name of the file to be downloaded must be defined here.

Session summary Returned identifier (ID) ID used as argument WF state is „running” WF state is „finished” „GetOutput” produces the „output.zip” file

Advanced features (1) Problem of resources requireing certificates: Solution: the Submit command can be extended by a certs tag. curl... - F …. where the zip file enumerats files with the following names x509up. and the contents of the „ x509up. ” files can be gained for example by the „ myproxy- logon” command issued on a Globus UIF machine: Example: myproxy-logon –s n40.hpcc.sztaki.hu –l MyAccountName –t 100 –o x509up.seegrid

Advanced features (2) A workflow definition may contain embedded workflow(s). Lets remember the syntax of portmappinng file: = / / OR = / The namespace separation permits the redefinition of files belonging to the eventual embedded workflows as well.